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ABSTRACT 



This guide, which is intended for volunteer literacy 
programs in Alberta, Canada, details a four- step process for evaluating 
volunteer literacy programs and using evaluation results for program 
development. The guide begins with an introduction, underlying principles and 
assumptions, and a glossary. Sections 1-4 detail the steps of the process, 
which are as follows: (1) getting ready for evaluation (planning and 

evaluation as part of the program cycle, forming an evaluation team) ; (2) 

focusing the evaluation (determining the evaluation's purposes, selecting 
standards, developing a work plan) ; (3) collecting information (reviewing 

current methods and relevant documents, completing observations, preparing 
and distributing data collection tools) ; and (4) interpreting and using 
research findings (compiling and summarizing the information, determining 
whether the program is achieving its standards) . Appendixes, constituting 
approximately two- thirds of the document, contain the following items: 10 
tables for use in applying the standards and evaluating program results; 
sample items for surveys, interviews, and focus groups; a case study based on 
the combined experiences of the seven literacy programs that pilot-tested the 
proposed evaluation method; background information on the Alberta Literacy 
Project Standards; good practice statements and standards; and a bibliography 
listing 45 references and suggestions for further reading. (MN) 
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Introduction 



Origins This document is the final product of the Alberta Literacy Program Stan- 

dards (ALPS) Project. ALPS was a project of the Association of Literacy 
Coordinators of Alberta, and was funded by the National Literacy Secre- 
tariat. Phase One of the project resulted in a set of good practice state- 
ments for volunteer literacy programs in Alberta, followed by a set of 
standards for those programs. Both the good practice statements and 
standards were developed in consultation with the province's literacy 
coordinators, who voted 98% in favor of approving them for use in the 
province's literacy programs. The purpose of Phase Two was to develop 
an evaluation process for programs to use for program development and 
accountability. As with previous steps, the evaluation process was devel- 
oped in consultation with the province's literacy coordinators. ALPS was 
jointly coordinated by Sharon Skage and Mamie Schaetti. 



intended audience 



This evaluation process is intended for volunteer literacy programs in 
Alberta. These programs, based in communities across the province, 
match trained volunteer tutors with adults who want to improve their 
reading, writing, math or spoken English skills. While specific goals and 
objectives are determined by each program, the following general objec- 
tives are held in common by all programs: 

• to recruit and train volunteer tutors, 

• to recruit and assess adult literacy students, 

• to match the tutors and students in pairs, 

• to support the pairs as they work one-on-one to achieve the students’ 
literacy goals, and 

• to promote the value of literacy in the broader community. 



In the few years that it has taken to develop this work, the way that liter- 
acy needs are addressed in Alberta has continued to evolve. More and 
more programs are incorporating a variety of approaches and services 
into their operations: family literacy, workplace literacy, and mentoring 
are just some of the ways in which community literacy programs are sup- 
porting the literacy development of Albertans. While the mandate of the 
Alberta Literacy Program Standards (ALPS) Project was to develop good 
practice statements, standards, and an evaluation process for volunteer 
tutor adult literacy programs, it is the authors’ hope that this process is 
designed in such a way as to be useful to plan and evaluate your entire 
program. In other words, it addresses the volunteer tutoring component — 
the common thread among all programs — but should allow for informa- 
tion about other services you might offer. 




U 
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Purpose 



Limitations 



Tips 




This document is intended to be used for program development and eval- 
uation. It is not intended to be used to compare one literacy program to 
another, or to be used as a job description for program coordinators. This 
evaluation process is intended to be used by the program staff with the 
support and participation of the program’s board or advisory committee 
and other program participants. It should be incorporated into day-to-day 
practice, rather than being used at the end of a program cycle. 

Because of the limited time and resources available to literacy programs 
for program development and evaluation, it is necessary for programs to 
adopt this evaluation process in increments, selecting specific areas of 
program practice to focus on according to their priority. 

Even without the need for adopting the process in increments, program 
staff may subconsciously or deliberately avoid evaluating some areas of 
their program’s operations. Interpretations of evaluation findings are sub- 
ject to the natural human tendency to look for evidence of what we 
already know or believe is true (Anderson 1987: 38). This can also be 
influenced by pressure to present the program in the best light in order to 
obtain funding and support. 

While third party or external evaluation is one way to address the issue of 
bias, this type of evaluation is not an option for most volunteer literacy 
programs. The design of this evaluation process, however, duplicates 
some of the elements of an external evaluation: the evaluation questions 
and the information needed to answer them have been developed in con- 
sultation with and approved by literacy coordinators across Alberta. 

This development and evaluation process is intended to be used by liter- 
acy coordinators and their programs with no previous experience in pro- 
gram evaluation. It should be pointed out, however, that the accuracy, 
validity, and reliability of the evaluation will increase with experience 
and with further training in qualitative and quantitative evaluation meth- 
ods. 

Despite these limitations, we feel that this process will be an effective 
tool in the maintenance and development of quality literacy programs. 

• Remember this is to be phased in, so choosing from among the evalua- 
tion questions and standards is vital. 

• Use it! Photocopy the worksheets and appendices, and write com- 
ments, highlight, illustrate, whatever it takes! 

• When in doubt, use the glossary. 
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Principles and Assumptions 



Principles 



Assumptions 



The following is a summary of what literacy coordinators in Alberta 

described in 1996 as the beliefs and values that they bring to their work. 

These beliefs and values are essential to understanding and interpreting 

the good practice statements, standards, and evaluation process. 

• Literacy programs should reflect and meet the needs of the individual 
in his/her context (home, work, community). 

• Accessible literacy education is a basic right. 

•• Literacy skills are empowering, and enhance a person’s quality of life. 

• Every individual has worth and ability, and deserves to be respected. 

• Learning happens when a person feels safe, is willing to take risks, 
and can feel personal success. 

• Learning is lifelong. We are all both teachers and learners. 

• Literacy is a community responsibility. 

• Literacy coordinators are facilitators, helping others so they can help 
themselves. 

• Honesty, sincerity, and fairness are essential values for a literacy pro- 
gram. 

The good practice statements, standards, and evaluation process are 

based on the understanding that: 

• Students are the first priority in all aspects of literacy programming 
and operation. 

• Volunteer literacy programs respect the diversity of experiences, back- 
grounds, and goals of participants. 

• Evaluation should be a participatory process; that is, participants 
should have the opportunity to take part in program evaluation. 

• Programs must receive stable and adequate funding in order to accom- 
plish and/or maintain good practice. 
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Glossary 



For the purpose of this document, the following definitions are used: 

Benchmark: a point of reference in measuring or judging quality. 

Community: the geographic area in which a program is located, whether 
it is as small as a neighborhood or as large as a municipal district, includ- 
ing the general population of that area; or more specific groups of indi- 
viduals and agencies within the larger population whose interests and 
activities are related to that of the literacy program. 

Evaluation: the process of collecting and analyzing information about a 
program or any of its components. Program participants use this informa- 
tion to make plans for the program, and to judge the effectiveness of the 
program relative to its stated goals. 

Exit assessment: the means of determining the extent to which a student 
has achieved his or her goals. (Also called final assessment.) 



Good practice statement: a description of an activity or condition that 
contributes to an effective, high quality literacy program. 



Indicators: key pieces of information used to indicate whether a pro- 
gram is meeting the standards of performance. 



Initial assessment: the means of determining an individual student's 
strengths and levels of competency in reading, writing and/or math, and 
his or her preferred learning style. Initial assessment information is used 
to develop an instructional program which meets a student's needs, inter- 
ests, and goals. (Also called placement assessment.) 



Mission statement: a brief statement which describes the purpose of the 
literacy program. 



On-going assessment: the means of determining a student's progress 
toward his or her goals. (Also called progress assessment.) 



Philosophy: guidelines and beliefs about how a literacy program should 
operate. A mission statement and a statement of values could be compo- 
nents of a program’s philosophy. 



Program: an individual volunteer literacy organization. When a standard 
says that “the program” is responsible for an activity or condition, the 
focus is on ensuring that the activity or condition takes place. Who actu- 
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ally carries out the task depends on the individual program; it may be the 
literacy coordinator, another paid staff, the board or advisory council, or a 
volunteer. The assumption is that this decision will be made at the pro- 
gram level. In this way, responsibility is not automatically placed on the 
coordinator if there are other program participants who take part in the 
program's operation. 

Program participants: those individuals directly involved in the pro- 
gram, including students, tutors, paid and volunteer staff, and advisory 
council or board members. 

Qualitative data: facts, claims, and assertions in narrative form, and not 
in numbers. 

Quantitative data: facts, claims and assertions presented in numerical 
forms. 

Reliability: the extent to which repeated measurements, using the same 
tool under the same conditions, produce the same results. 

Sources of information: the people, records, reports, or other materials 
that provide indicator information. 

Staff: both paid personnel and the volunteers who support the program in 
ways other than tutoring. 

Standard: a benchmark against which programs can be evaluated. 

Validity: the extent to which a tool measures correctly what it is designed 
to measure. 

Values statement: a summary of the ethical principles which underlie 
program practice. 

Vision statement: the picture that program participants develop of their 
organization in the future, describing the ideal to which their program 
aspires. 

Volunteer literacy programs: the provision of student-centered literacy 
education services to communities through the use of volunteer tutors . 
Although services may frequently include broader social and communi- 
cation skills, the focus is on improving reading, writing and math skills. 

Volunteers: those who donate time and energy to help achieve the objec- 
tives of the literacy program, including tutors, volunteer staff, and advi- 
sory council or board members. 
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Step I 


Getting Ready for Evaluation 


Purpose of Step 1 


In Step 1 , you will become familiar with this document and the concepts 
of planning and evaluation; recruit people from your program and com- 
munity as members of an evaluation team; think about and describe the 
characteristics of your program; and look at how ready your program’s 
participants are for an evaluation. 


Why it's useful 


When this document was piloted in seven literacy programs around the 

province, coordinators reported that they found Step 1 useful because: 

■ putting an evaluation team together was a pivotal part of using the 
evaluation process, and it also made the coordinator take the time to 
plan and organize the process; 

■ the list of responsibilities helped in putting the evaluation team 
together; 

• having to explain the process to other potential evaluation team mem- 
bers made the coordinator become very aware of the process right 
from the start; 

• it made the coordinator collect important information and get it to the 
evaluation team; and 

■ describing the program’s context helped the coordinator to think about 
what the program is doing, what it is not doing, and where they might 
focus the evaluation. 

For an example of a program’s experience in using Step 1 , please see the 

case study on page 144. 



Holv much time does it take? 

On average, it took the pilot site programs five hours to work through 
Step 1 . This included reading the document, calling potential evaluation 
team members, meeting with the team, and locating and recording infor- 
mation about the program. 
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Becoming familiar with this document 

We encourage you to spend some time becoming familiar with this docu- 
ment before working through Step 1 . Taking a few moments to read the 
Introduction, Principles and Assumptions, and the Glossary in the open- 
ing pages will provide a context for using the tool. Reviewing the first 
page of each of the four steps will help you to understand what the pur- 
pose of each step is, how others have found it useful, and how the process 
moves from one step to another. 

The rest of the document contains appendices, and again there is an intro- 
duction for the first two explaining their purpose and how others have 
found them useful. The third appendix is a case study that describes a lit- 
eracy program’s experiences in using this tool. The last three appendices 
are self-explanatory, containing background on the project that produced 
this document, the previously-published good practice statement and 
standards document, and a bibliography. 

The bulk of this document may be intimidating at first, but you’ll quickly 
find that the program development and evaluation tool itself is quite sim- 
ple and straightforward. Much of the document is made up of appendices, 
which contain very detailed information provided to make this task as 
easy as possible for coordinators and their programs, while still allowing 
for the flexibility and diversity so important to literacy work in our com- 
munities. 



Planning and evaluation as part of your program cycle 

Planning and evaluation are part of a continuous cycle that is an integral 
part of what happens in literacy programs. At its most informal, it may be 
a matter of the coordinator thinking, “What are we going to do this 
year?” and asking tutors, “How is it going?” This document is designed 
to address the need expressed by literacy coordinators for a tool that will 
support planning and evaluation, both for program development purposes 
and for accountability. 




summing up 
the results of 
your program 



using 

information 
in program 
planning 



PROGRAM CYCLE 



N . 

monitoring 
your program 
and making 
necessary 
changes 
X 



Looking at this cycle, we see that evaluation is done for two general rea- 
sons. The first is to monitor, on an on-going basis, how well your pro- 



gram is operating. The second general reason evaluation is done is to sum 
up the results of your program’s activities and use the information to 
report on program results and accomplishments. 



Information from both of these types of evaluation - day-to-day monitor- 
ing and summing up — is used in planning for future programming. Mon- 
itoring your program will give you information that you can use to 
identify and address any problems that may come up, to know what is 
working well, and to improve services to students. Looking at the results 
of your program at the end of a cycle will give you information that you 
can use as you and your council or board make plans for the upcoming 
year. 



Forming an evaluation team 



One of the first things you should do to get ready for evaluation is to form 
an evaluation team. In some instances, the program coordinator may need 
to take sole responsibility for evaluation, but usually there are other peo- 
ple who can share the responsibility. 



The evaluation team should be quite small, perhaps four or five people, 
and include representatives from the program’s major stakeholders: stu- 
dents, tutors, paid and volunteer staff, advisory council or board mem- 
bers, and community partners. When thinking about whom to approach, 
consider what knowledge and/or skills the team should possess. For 
example, you might be looking for people who are familiar with the pro- 
gram, who have good interpersonal skills, or who have some experience 
with basic research, such as doing focus groups or interviews. 



Inviting participation We suggest that you send a letter out to prospective evaluation team 

members. (You may wish to have a tutor go over the letter with a stu- 
dent.) In the letter, give a brief description of the literacy program, the 
need for evaluation, and a summary statement of the role of the evalua- 
tion team. Emphasize the importance of involving members of the pro- 
gram and the community in planning and evaluating the program, and ask 
them to consider participating. If possible, give a start date and an end 
date for their involvement. End the letter by letting them know when you 
will be contacting them. (There is a sample letter to potential evaluation 
team members on the following page.) 

Follow up on the letter with a telephone call or conversation with poten- 
tial team members. Ask them if they are interested in being part of the 
evaluation team, and if they are, invite them to a meeting to learn more 
about the evaluation process and about the role of the evaluation team. 
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Sample letter to potential evaluation team members 

(Date) 

Dear , 

The ABC Literacy Program is committed to serving literacy students in Anywhere and surrounding 
areas. As a (tutor/volunteer/student/advisory council or board member/supportive business person), I 
know that you share that commitment. 

Since the program began nine years ago, our purpose has been to help community members improve 
their literacy and English as a Second Language (ESL) skills. Students come from Anywhere and the 
surrounding area to work one-on-one with our trained volunteer tutors and we do our best to help 
them meet their literacy or ESL goals. 

One way we can make sure that we are doing our best for students is by evaluating our program. We 
already do this all the time in an informal way. For example, tutors ask students how the lesson went, 
or I ask tutors what kind of in-service workshop they might want. We can also ask this kind of ques- 
tion more systematically and that's what we plan to do this year. We will decide what questions we 
want to have answered about the program and how it works. Then we will ask many different partici- 
pants in the program for their views. We will make decisions about the program and base future plan- 
ning on what we have found out. 

An evaluation team will lead this work. I am writing you to ask you to be part of that team. The eval- 
uation team will meet five or six times over the course of the year. The exact number of meetings will 

be decided by the team at the first meeting on . We'll also discuss at that meeting what 

the specific role of the committee and its members will be. We intend to finish the evaluation by 



I realize that you already give a lot to the program and that you are a busy person. I am writing you 
because your knowledge of the program and your perspective on it are just what we need. We will be 
asking others to participate who see the program from different points of view. The team will have 
total of four or five people on it. I will be a member and I hope to have you and (a student/a tutor/an 
advisory council or board member/someone from the community at large) join me. Together, we can 
take a good look at the program, see where our successes lie and where we might do better. 

Please, do consider joining the team. I hope the experience will be rewarding and I'm sure that we will 
all leam a lot. 

If you have any questions, please call me at the office. I'd be happy to provide more information. 
Sincerely, 

Jane Doe, Coordinator 



Setting the Compass 



page 1 1 



Orienting the evaluation team 



When you first meet with members of your evaluation team, you will 
need to introduce and discuss the concept of evaluation, as well as the 
process that will be used. 

Evaluation of a literacy program serves two main purposes: to gather 
together information that can be used for program development (“to see 
what works and what we could do better”), and to collect information 
that can be used for accountability and demonstrating the value of the 
program. You will find more information that will be helpful in introduc- 
ing the concept of evaluation in the “Introduction” section earlier in this 
document. 

To give the evaluation team a brief overview of the program development 
and evaluation process, make copies of the Table of Contents from this 
document and use that during your meeting. You can also use the infor- 
mation in Appendix D to give the team a sense of how this process was 
developed and how literacy coordinators were involved. 

What the evaluation team as a whole and what individual team members 
will do during the evaluation will depend on your specific situation. In 
your first meeting, however, you may want to consider the following as 
possible general responsibilities for the evaluation team: 

• ensuring a common understanding of the purpose of evaluation, such 
as what can and cannot be expected from evaluation; 

• ensuring common frames of reference, such as common definitions of 
terms, agreement on timelines and resource expenditures, etc.; 

• step-by-step planning of the evaluation (see the workplan in Step 2); 

• deciding who will lead or coordinate the evaluation; 

• securing the required resources; 

• monitoring the progress of the evaluation; and 

• suggesting changes to the evaluation process if necessary. 

When discussing the evaluation team’s responsibilities, be very clear 
about expectations for their involvement. For example, the evaluation 
team will likely meet up to six times in the first year/cycle that the evalu- 
ation process is used. Similarly, people may have certain expectations 
about their involvement, and it is important for those to be part of the dis- 
cussions as well. 

Once your and your evaluation team begin working with the program 
development and evaluation process, you will be able to make decisions 



as to what specific role the team and its individual members will play in 
carrying out the four steps of the evaluation. 

Describing your program and its context 

Before you can effectively evaluate your program, you need to record 
information about your program and its context. This information will be 
used later when you are interpreting the information collected in the eval- 
uation. For example, your evaluation results may show that the coordina- 
tor is not contacting students and tutors as often as the standards suggest. 
Looking at the human resources available (question #3 below), however, 
it may be clear that the hours of paid staff time per week would need to 
increase before that contact could happen. In other words, looking at the 
program and its context can help to explain your evaluation findings. 

This information may already exist in the form of planning records, 
annual reports, funding proposals, etc. If not, you may find the following 
worksheets helpful in collecting the information. 

Some of the information requested may not change from year to year, as 
in “size of area,” and “number of months per year the program operates.” 
Where information is time sensitive, use information from the program 
cycle being evaluated unless otherwise noted. For example, use your 
funding figures for 1999 - 2000 if that is the program cycle you are evalu- 
ating. 

Instructions: Photocopy and complete the questions on the following six 
pages. The first two pages are both a “pull-out” to send to Community 
Programs as well as part of this process of reflecting on your program 
and its context. (Be sure to save a copy of the completed “pull-out” 
sheets for your records.) 

If applicable , the questions can be divided among members of the evalua- 
tion team to complete. 

During our meetings with literacy coordinators, we were told fre- 
quently that they would prefer not to have to use this document and 
complete a separate report for Community Programs of Advanced 
Education and Career Development. We have, with the govern- 
ment’s cooperation, incorporated the “Participation Summary” form 
into this document. Use pp. 14 - 15 and pp. 92 - 96 (in the Program 
Results section) to replace the Participation Summary from Commu- 
nity Programs. (You will receive a letter from Community Programs 
advising you when to fill in and submit the forms.) 
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Volunteer Tutor Literacy Programs 
Participation Summary 



Program cycle being evaluated: 

(e.g. Sept. 1/99 - June 15/00 or Apr. 1/99 - Mar. 30/00) 

Prepared by: Telephone number: 

Program name: 

Community Adult Learning Council: 

Name specific communities/areas where students were tutored during this program cycle (e.g. Sandy 
Creek and rural areas of Wild Rose, Briarlea, and Round Lake): 



Population of area: 

Size of area (estimate square kilometres of total area, or use average radius from program 
office): 

Total Community Programs grant allocation for volunteer tutor program: $ 

Total additional revenue for volunteer tutor program (see page 98 when completing #8 in Program 
Results): $ 

Sources of above additional revenue: 



(Submit pp. 14 - 15 and pp. 92 - 96 to Community Programs.) 
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Relationship to other organizations, if applicable (e.g. “XYZ College hires staff, administers funding, 
and provides a board that directs the program,” “ABC Learning Council hosts the program and has a 
representative on the advisory committee”): 



Legal or non-legal status of literacy program (e.g. “literacy program is an incorporated society,” 
“literacy program operates under the local school division”): 



What are the literacy program’s goals for this program cycle? (e.g. “At the recent planning day, goals 
for this program year included increasing the number of student/tutor pairs by 5, improving access to 
the program, and raising our community profile.”) 



(Submit pp. 14 - 15 and pp.92 - 96 to Community Programs.) 
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1 What are the program's features? 



Information needed 


Description 


geographic location (e.g. 
County of Lethbridge, north- 
east Edmonton) 




size of program (list sepa- 
rately the number of students, 
tutors, and other volunteers at 
the end of the last program 
cycle) 




number of months per year the 
program operates (do not 
include months the program is 
closed but pairs continue to 
meet) 




length of time the program has 
been in operation 




other (e.g. program philoso- 
phy, description of facility) 




Note: before adding other 
information here, check the 
remaining questions in this 
section and the "Program 
Results” questions on pp. 91- 
99 to avoid duplication. 
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2 What are the human resources available in this program cycle? (Infor- 
mation about volunteers is included in the “Program Results” ques- 
tions on pp. 91 - 99.) 



Information needed 


Description 


number of paid staff (identify 
full-time and part-time) 




hours of paid staff time per 
week 




number and type of external 
resource people available to 
the program (e.g. program 
development support, counse- 
lors, trainers) 




total hours of training and pro- 
fessional development for paid 
staff (include program-spe- 
cific, regional, provincial) 




other (e.g. part-time or full- 
time contract staff, estimated 
unpaid overtime) 
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3 What are the other resources available in this program cycle? 



Information needed 


Description 


grants or project funding for 
services, projects, etc. other 
than the volunteer tutor pro- 
gram 




contributions in kind (e.g. 
office space, telephone, use of 
photocopier) 




resource library (list general 
contents, e.g. videos, books, 
games, etc.) 




professional associations (e.g. 
Literacy Coordinators of 
Alberta, Alberta Association 
for Adult Literacy) 




other 





( Photocopy and save original) 

Gouging rcodincss A program’s readiness to conduct an evaluation depends on many factors, 

including time, money, attitudes, and leadership. Before proceeding with 
an evaluation, it would be useful to determine just how ready your pro- 
gram is. Ideally, you should have board/council/committee members and 
staff answer the following questions independently so you can see areas 
of agreement and disagreement as well as issues that will act as barriers 
to evaluation. If there are significant disagreement or barriers, steps will 
need to be taken to address these before a successful evaluation can be 
implemented. 

Instructions: Photocopy the checklist on the following page and use it to 
collect information on how ready your program is for evaluation. The 
pages following the checklist will give you ideas on how to address dis- 
agreement or barriers. 
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Checklist to see how ready our program is for evaluation 

Part of preparing to do program evaluation is finding out how ready our program is for evaluation. To 
help us to find out if there is anything that needs to be addressed first, please complete the following 
checklist by answering “yes” or “no” for each question. You do not need to include your name, but 
please tell us whether you are a staff person, board/council/committee member, volunteer, or student. 
Thanks! 





Yes 


No 


Don’t 

know 


1 . Do people involved with the program agree on the mission, goals, and 
major activities of our program? 








2. Is conducting program evaluation seen as consistent with the program’s 
philosophy, or the approach we take to literacy work? 








3. Are people connected with our program interested in evaluation? 








4. Is evaluation seen as providing useful information to our program? 








5. Do people see the benefits of evaluation as being greater than the costs of 
time and money? 








6. Is program evaluation seen as a threat and a sign of distrust? 








7. Does our program have a planning structure that will use the evaluation 
findings? 








8. Do staff, students, and volunteers believe that their ideas and feelings are 
valued and respected by program decision makers? 








9. Is there money available to support an evaluation of this program? 








10. Is there staff time available for the evaluation? 








1 1 . Are staff willing to take risks in trying new ways of doing things? 








12. Do leaders in our program demonstrate excitement and energy about 
change and innovation? 









I am a... staff person tutor or other volunteer 



board/council/committee member student 
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Addressing barriers to evaluation 



If the results of the checklist on the previous page tell you that there are some significant barriers or 
challenges to doing program evaluation, the following ideas should help you to address them. They 
are organized according to the items on the checklist. 



Barrier 


Suggestions for addressing the barrier 


1 . There is a belief that people 
do not agree on the mission, 
goals, and major activities of 
the program. 


People need to agree on what to evaluate before evaluation can 
be done, or the findings will be meaningless. If using this check- 
list shows that people don’t agree on what the mission, goals, and 
major activities of the program are, these need to be reviewed 
and communicated to participants. If there isn’t a mission, devel- 
oping one should be a priority. 


2. Conducting program evalua- 
tion is not seen as consistent 
with the program’s philoso- 
phy or approach. 


Program evaluation is about maintaining and improving quality 
services to students. You may hear statements like “Volunteers 
shouldn’t be asked to complete surveys; it’s enough that they 
give their time tutoring,” or “We’ll scare students away by asking 
too many questions.” If evaluation is presented as a means of 
improving services to students, it may be recognized as being 
more consistent with the program’s approach. An open discus- 
sion of how evaluation is seen as consistent or inconsistent may 
provide valuable suggestions for the best way to conduct the 
evaluation. 


3. People connected with the 
program are not interested in 
evaluation. 


If people are not interested in evaluation, perhaps they don’t 
understand the relationships between evaluation, planning, and 
delivering quality programs. The benefits of program evaluation 
(to recruit and retain staff, enlist and motivate volunteers, attract 
new students, engage partners, retain or increase funding, raise 
public awareness, etc.) should be discussed with program partici- 
pants. 


4. Evaluation is not seen as 
providing useful information 
to the program. 


Discussing the information that evaluation will yield, using the 
standards for volunteer literacy programs developed by the 
L.C.A. as a reference, might be helpful in demonstrating that 
evaluation is not just required for accountability but useful for 
program development. 


5. People do not see the bene- 
fits of evaluation as being 
greater than the costs of time 
and money. 


Evaluation does take time and money to accomplish. Having 
people identify what benefits would be worthwhile might encour- 
age them to see the program invest its resources in evaluation. 
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Barrier 


Suggestions for addressing the barrier 


6. Program evaluation is seen 
as a threat and a sign of dis- 
trust. 


Discussing what program evaluation is not for might be helpful 
in alleviating concerns. It is not an evaluation of the program 
coordinator, and it is not for the purpose of comparing one pro- 
gram to another. Evaluation is part of program operations in most 
fields, and is now expected of literacy programs. Emphasis on the 
use of evaluation findings in program planning and development 
will be useful. 


7. The program doesn’t have a 
planning structure that will 
use the evaluation findings. 


If evaluation findings are not used, concerns about wasting time 
and resources will be confirmed. If the program doesn’t currently 
have a planning structure in place, now is the time to develop 
one. It may be as simple as an annual planning day involving 
staff, board/council/committee members, tutors, and student. 


8. People do not feel their ideas 
and feelings are valued and 
respected. 


Evaluation involves hearing from people about what effect the 
program is having. People will be encouraged to give their views 
on the program if they feel the interest is genuine and their com- 
ments will be used. 


9. There isn’t money available 
to support an evaluation of 
this program. 


This is not an easy one to address. The amount of money neces- 
sary for evaluation will depend on how extensive an evaluation 
your program wants to do. It will include money for postage if 
you are mailing surveys, some amount of photocopying, and 
money for refreshments if you are bringing people together for 
focus groups and evaluation-related meetings. It may include 
long distance charges depending on your methods of collecting 
information and where your program participants live. Money for 
evaluation should be included in the program’s grant application, 
and should be an ongoing line item in the program budget. 


10. There isn’t staff time avail- 
able for the evaluation. 


As with #9, the amount of staff time depends on the amount of 
evaluation. Again, time for evaluation should be included in the 
budget calculations for staff wages. In addition, are there creative 
ways to reallocate some staff responsibilities to other program 
participants so as to make time available for evaluation, e.g. pro- 
ducing the program’s newsletter? (When this tool was piloted, 
the average amount of time needed to conduct the entire evalua- 
tion process, looking at a minimum of 10 standards, was 40 
hours.) 
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Barrier 


Suggestions for addressing the barrier 


1 1 . The staff are reluctant to 
take risks in trying new 
ways of doing things. 


Introducing this evaluation process may mean a change in how 
things are done in the program. How have staff reacted to change 
in the past? Do they feel confident in taking risks, or have they 
been discouraged in the past from trying new things? Staff need 
to feel supported in order to feel comfortable with taking risks. 


12. Leaders in the program do 
not demonstrate excitement 
and energy about change 
and innovation. 


This again relates to how people in the program react to change. 
Leaders — whether board members or the program coordinator — 
should be positive role models, taking a positive attitude towards 
the changes that this evaluation process will bring. 
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Step 2 


Focusing the Evaluation 


Purpose of Step 2 


In Step 1 , you prepared some background information on your program 
that will help you to interpret your evaluation findings. You also looked at 
some suggestions for finding out how ready your program is to be evalu- 
ated. In Step 2, you will identify the reasons why you are doing an evalu- 
ation, and who you are doing it for. You’ll use that information as you 
choose questions that you need to answer about your program, and from 
there you will select standards that relate to those questions. 

Once you have decided upon the standards for this evaluation cycle, you 
will look at what resources you’ll need to carry out this evaluation. You’ll 
develop a workplan for the evaluation, and look at how you can best 
approach people to let them know that these activities will be taking 
place. 


Why it's useful 


When this document was being piloted, coordinators reported that they 

found Step 2 useful in the following ways: 

• without a real focus of why we’re doing evaluation and what we really 
want to know, it is wasted time and energy (something literacy 
projects stretch to the limit already); 

• the process of thinking through who, what, when, and how was very 
helpful in starting to plan the evaluation; 

• it allowed them to select standards that will benefit and improve pro- 
gram operation; 

• having the lists of indicators, possible sources of information, and sug- 
gested methods of collecting information already prepared was very 
helpful; 

• the “Resources and Priorities” section really helped to get a clear and 
realistic picture of what they needed to do and how they could go 
about their work; and 

• working through this step really helped to understand the evaluation 
process. 

For an example of a program’s experience in using Step 2, please see the 

case study on page 144. 



How much time does it take? 

On average, it took the pilot site programs a total of six hours to complete 
Step 2. 
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Clarifying why you're doing evaluation, and for whom 



In Step 1 , we looked at how evaluation happens in an on-going way when 
you monitor your program’s operations, and as a way of summing up the 
results of your program’s activities. It is unlikely that you will want to 
systematically evaluate every aspect of your program at once, so this sec- 
tion will help you to focus the evaluation by thinking about what infor- 
mation will be needed and by whom in this program cycle. 



Whether you’re doing evaluation for the purpose of monitoring your pro- 
gram operations or looking at what the program results are, you will 
likely have different audiences for different parts of your evaluation find- 
ings. For example, looking at how well initial tutor training went would 
be useful to staff before they plan a follow-up workshop. Information on 
how many students were served by the end of the program cycle would 
be useful to a number of different audiences, such as partner agencies and 
funders. An important part of focusing your evaluation is to think about 
the different audiences that you need to provide with information. 



In looking at the following suggestions for audiences, you may think that 
all of them are important and necessary. It may not be feasible, however, 
to try to provide every stakeholder with information in one program eval- 
uation. You may need to set priorities for this evaluation, and incorporate 
the rest into program planning for following years. 



Instructions: The checklist below is provided to help you focus on why 
you are doing evaluation, and who you need to provide evaluation infor- 
mation to. There are a few examples given to suggest why these groups 
might be important audiences. Photocopy the checklist and rank your 
choices in terms of priority if you need to. After doing this step you will 
move on to looking at what kinds of questions these audiences might have 
about the program, and how those questions could be answered. 



Monitoring program operations 

• paid staff (e.g.for short-term program planning) O 

• students (e.g. to provide encouragement) Q 

• tutors (e.g. to show that the program is a worthwhile place to volun- 
teer) Q 

• board/council/committee members (e.g. to provide direction to staff 

and other program participants) Q 
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□ 

□ 

□ 



• funders (e.g. for interim reports) 

• other 



Determining program results 

• paid staff (e.g. for long-term program planning, recruitment) □ 

• students (e.g. to show that the program helps students achieve their 
goals) 

• volunteers (e.g. to confirm the value of their contributions) G 

• board/council/committee members (e.g. to make decisions regarding 

allocation of program resources) Q 

• funders (e.g. to demonstrate accountability and value of service) Q 

• partner agencies (e.g. to help make decisions about their relationship 

to the program) Q 

• community (e.g .to gamer community support for the program) □ 

• other Q 

• □ 

(Photocopy and save original) 



Identifying questions you need to answer about your program 

In the previous section, you identified program participants and other 
stakeholders who will need information from this program evaluation. In 
this section, you will think about the kinds of questions that these people 
have about the program. These questions have been divided into two 
types: those that deal with program operations and those that deal with 
program results. 

“Program operations” questions deal with how well the program is being 
implemented. This section contains the standards for volunteer literacy 
programs developed and approved by Alberta’s literacy coordinators. 
(For information on the LCA’s Good Practice Statements and Standards, 
see Appendix E.) 
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“Program results” questions relate to the results of the program and its 
impact on participants. For example, program results include the number 
of hours of tutoring provided and the number of people served by the pro- 
gram. 



As mentioned earlier, the amount of time and resources available for 
evaluation differ from one program to the next. Depending on your pro- 
gram’s situation, and your current evaluation practices, you may choose 
to answer all of the program operations and program results questions, or 
you may select those that are a priority for this program cycle. 



Instructions : Photocopy and review the following list of questions for 
program operations and program results. Check off those questions that 
are the highest priority for this program cycle. In the next section you 
will select from related standards. 



Program Operations 



1. In what ways is our program community based? Q 

This question relates to standards on assessing and responding to 
community needs; relationship with board/council/committee; 
community partnerships and support; and promotional activities. 




2 . 



In what ways does our program have a relevant and applied philoso- 
phy? Q 

This question relates to standards on the development, review, 
application, and communication of philosophy-related state- 
ments. 



3. In what ways do we deliver programming that addresses the needs of 

the individual? Q 

This question relates to standards on intake procedures, includ- 
ing interview, initial assessment and orientation; student goal set- 
ting and assessment; instruction, including matching and 

working with a tutor; student support; and student recognition. 

4. In what ways are we providing quality tutoring? □ 

This question relates to standards on recruitment and interview; 
initial and on-going training; tutor support; and tutor recognition. 

5. In what ways do we have sound program administration? □ 

This question relates to standards on accountability and responsi- 
bility (e.g. decision making, operational responsibility, fiscal 
management); recruiting and hiring staff; staff training and sup- 
port; confidentiality and record keeping; effective use of time 
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and resources; volunteer recognition; and resource materials, 
facilities and equipment. 

6. In what ways is our program effectively planned and evaluated? □ 
This question relates to standards on program planning and pro- 
gram evaluation. 

Program Results 

7. What services were delivered during this program cycle? Q 

This question has to do with the number of people involved in 

the program; hours of participation; and other services provided. 

*Note: #7 is required for all programs as part of the Participation 
Summary-related information for Community Programs. For that rea- 
son, it also includes student satisfaction items. 

8. What were the costs of delivering the program? Q 

This questions relates to actual costs of running the program as 

well as other contributions. 

9. What is the impact of the program on its participants? □ 

This question has to do with benefits to or changes in participants 

as well as tutor satisfaction with the program. 



(Photocopy and save original) 



“After reading through the questions, I chose the ones that I felt 
would be most beneficial for our program. Certain aspects of our 
program had concerned me, and there were questions that addressed 
these concerns.” 

— pilot site coordinator 
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Selecting 



standards and related items 



In the previous section you reviewed a list of questions, and selected 
those that are the most relevant for this evaluation. In this section you will 
review the standards related to the Program Operations questions, and 
select those that you feel would be the most relevant and feasible for this 
evaluation. 



The introduction of standards should be a developmental process. 
The achievement of standards, as well as being able to demonstrate 
the achievement of standards, will take time for some programs. The 
important thing is to be working toward achieving standards in a 
manageable way and as time and resources permit. Also, remember 
that the purpose of evaluation should be to demonstrate the program's 
successes as well as to identify areas that need development. 



Instructions: Using your list of selected questions as a guide, turn to 
Appendix A on page 62. Photocopy the relevant pages and save the orig- 
inals for future use. For each evaluation question that you have selected, 
review all the related standards. Select the standards that you think 
would be the most useful and feasible to answer the evaluation questions 
that you have selected, considering your goals for this year and the audi- 
ences for your evaluation findings. 
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called “other.” This is for adding your own items that you feel would 
help to answer the evaluation question. (See pp. 100 for an example of 
how this is used.) 



Once you have selected the standards, you will need to decide how you’ll 
show where your program is at in relation to the selected standards. Use 
the tables to select indicators, sources of information, and methods of 
collecting information. Remember that there is often a range of indica- 
tors suggested, and your program is to choose from among them unless 
otherwise noted. 



Remember that using 2-3 sources of information and collection meth- 
ods will provide a more valid and comprehensive picture of the literacy 
program than relying on a single source and/or method. For example, you 
will get a better sense of things if you don’t limit your evaluation to 
reviewing documents, but ask students and tutors for feedback through 
surveys, interviews, or focus groups. Before selecting the methods of col- 
lecting information, see page 38 in Step 3 for tips on how surveys, inter- 
views, etc. can be used. 
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When using a particular method of collecting information, make it worth- 
while in terms of the number of standards it can address. For example, if 
you want to survey tutors to find out how satisfied they are with their 
tutor job description, you should select other standards where you can 
collect information by surveying tutors. 



“I didn’t find the standards overwhelming or unattainable even though 
we are a small project.... I have discussed this with several people and 
often I hear comments such as, ‘That’s fine for the city but what about 
our little country program?’ Well, I really feel that whether you have 
10 or 200 students or 1 or 20 staff, the standards can be met....” 

— pilot site coordinator 



Selecting Program Results items 

When you selected evaluation questions earlier in this step, you will have 
noticed that there are three questions that address Program Results, rather 
than Program Operations (standards). Of these three questions, question 
#7 is to be used by all programs to replace the Participation Summary 
formerly sent out by Community Programs at Advanced Education and 
Career Development. (Community Programs will send a letter letting you 
know when to fill out and submit those pages.) You’ll find pull-out sheets 
for question #7 at the back of Appendix A (pp. 92 -96) and instructions 
on filling out Participation Summary-related information on pp. 13 and 
91. 

There are also worksheets at the back of Appendix A to use as you select 
items related to questions #8 and 9, if those were among your choices for 
evaluation questions earlier in this section. 

Instructions: Considering your choices for Program Results evaluation 
questions on page 27, review the worksheets at the back of Appendix A 
(pp. 92 - 99. If you chose evaluation question #8 or 9, select the table 
items that would be useful for this planning and evaluation cycle. Where 
necessary, select from the sources of information and methods of collect- 
ing information as you did earlier in Appendix A. 
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Identifying resources and setting priorities 

Now that you’ve decided how you want to focus your evaluation for this 
year, you need to look at what resources would be needed to carry it out, 
and at what resources you actually have available. (There is a worksheet 
on the following page to help you do this.) If there is a significant differ- 
ence between the two, you may need to revise your evaluation plans and 
establish further priorities. 

In determining the resources you’ll need for your evaluation, you’ll need 
to look at some basic resource needs as well as resource needs related to 
your selected evaluation questions and standards. Basic resource needs 
might include staff time to organize an evaluation team; costs for your 
evaluation team to meet; staff time to compile information collected; and 
staff time to interpret the information, draw conclusions, and recommend 
follow-up actions. There may also be some time and copying costs 
involved in getting the findings out to program stakeholders. (The amount 
of paid staff time for these tasks depends on how involved volunteer eval- 
uation team members are in the evaluation.) 

Resources needed for addressing selected evaluation questions and the 
standards related to them depend on your choices. You may have costs 
associated with bringing participants together for focus groups, including 
refreshments, renting a meeting place, etc. You may need a resource per- 
son (board member, agency representative, etc.) not known to students 
and tutors for conducting interviews or holding focus groups. Staff or an 
evaluation team member will need time to review documents. Photocopy- 
ing may be necessary for making copies of questionnaires and survey 
forms. 

Once you have calculated the cost of evaluation in financial terms, be 
sure to include evaluation as a line item in your program’s budget and 
funding applications. 

If the chart on the following pages shows you that you do not have the 
resources required for all of the evaluation questions and standards that 
you’ve selected, you will either need to find the resources or assign prior- 
ities to your selections, and leave the rest until another evaluation cycle. 
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Resources and Priorities 

Instructions: Photocopy these pages and, according to your selections earlier in this section, list the 
standards and evaluation questions by number in the following chart. ( List them by number, write 
them out, or cut and paste from Appendix A.jWrite down the resources you’ll need for each one, 
according to your plans for sources of information and methods of collecting information in Appendix 
A. In the third column, check whether these resources are available or not. After you’ve finished, 
assign priority if necessary. 



List evaluation 
questions and 
standards 


List resources needed 


Are resources 
available? 
(Yes/No) 


Priority 
for this 
year? 


example: 4.2: The 
program has 
clearly written and 
realistic job 
descriptions for its 
tutors. 


• time to review job descriptions (10 minutes) 

• time to prepare a tutor survey and include it 
in the January newsletter, using questions in 
Appendix B (1 .5 hours) (Note: this time would 
allow for any other standards requiring an 
item in the tutor survey, so you would repeat 
this for resources needed, but put “0” for the 
time required.) 


Yes 

Yes 




example: # 7 : stu- 
dent satisfaction 
questions 


• time to prepare a set of questions for a focus 
group with students, using questions in 
Appendix B (.5 hour) 

• time to invite students to meeting ( 1 hour) 

• time for meeting (2 hours) 

• money for refreshments 


Yes 





















( Photocopy and save original.) 
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List evaluation 
question by 
number or 
standard by number 
and letter 


List resources needed 


Are resources 
available? 
(Yes/No) 


Priority 
for this 
year? 



















































( Photocopy and save original.) 
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Developing a workplan for the evaluation 



The next step in the evaluation process is to develop a workplan: what 
needs to be done, by whom, and when. Such a workplan will help you to 
plan ahead and to incorporate evaluation into your everyday practice. We 
have included a sample workplan on the following pages for you to use 
and adapt as necessary. It uses an example of a September 1 to June 30 
program cycle. 



In setting the timelines for the evaluation, the evaluation team should 
keep several considerations in mind. 



• First, evaluation of the literacy program must serve both the program’s 
on-going internal need for information and its cyclical need to report 
to funders. 

• Second, the literacy program’s own cycle, deadlines, and planned 
events will affect the evaluation timelines. For example, it might be 
wise not to plan a labour-intensive portion of the evaluation process to 
coincide with tutor training. 

• Third, the availability of resources, especially from outside the pro- 
gram, may be limited at certain times. For example, if a program were 
to ask an Adult Basic Education instructor to lead a focus group with 
students, consideration would have to be given to that program’s 
schedule. 



Finally, agencies and organizations which support the program will 
require reports to be submitted according to the deadlines established 
by those organizations. 



The evaluation team should realize that any timelines established at this 
stage may shift as the evaluation is carried out, and that dealing effec- 
tively with emerging issues is more important to the success of the evalu- 
ation than meeting deadlines which were established before the issues 
were known. 
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Sample Workplan 



Instructions: Complete the dates for each activity and the person(s) responsible. Shift activities to 
other months and add your own activities as necessary. 



Timeline 


Activity 


Person(s) responsible 


September 








form an evaluation team (Step 1) 






collect information on students entering the pro- 
gram (ongoing) 










October 








complete this workplan 






complete worksheets on program context (Step 1) 






complete exercise on program readiness (Step 1) 






clarify why you’re doing evaluation (Step 2) 










November 








choose evaluation questions; select standards and 
related items; identify resources needed and avail- 
able; and set priorities if necessary (Step 2) 






let program participants know what their roles 
will be in the evaluation (Step 2) 






prepare surveys, questionnaires, etc. to use in col- 
lecting information (Step 3) 










December 








test surveys, questionnaires, etc. (Step 3) 










January 








begin to review documents (Step 3) 











(Photocopy and save original.) 
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Timeline 


Activity 


Person(s) responsible 


February 








finish reviewing documents (Step 3) 










March 








conduct observations (Step 3) 






distribute surveys (Step 3) 










April 








report relevant results to Community Programs 






compile results of surveys, observations (Step 4) 






hold interviews (Step 3) 






hold focus groups (Step 3) 










May 








summarize information collected (Step 4) 






interpret information and relate it to standards and 
evaluation questions (Step 4) 






develop conclusions and recommended actions 
(Step 4) 






inform students, tutors, etc. of evaluation findings 
and any follow-up actions (Step 4) 










June 














(Photocopy and save original.) 
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Informing people about the evaluation 



Once you have your workplan completed, you will need to inform pro- 
gram participants about what their role will be in the evaluation. Those 
who will be contacted to give feedback on the program should be aware 
of this as early in the program cycle as possible. The idea of participating 
in the program evaluation should be presented as a routine aspect of pro- 
gram practice, not something separate from it. 

For example, you might want to put a note in the tutor newsletter advis- 
ing them that they will be receiving a survey in March to gather their 
ideas on the program. 
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Step 3 Collecting information 



Purpose Of Step 3 In Step 2, you focused your evaluation by selecting evaluation questions, 

choosing standards, looking at the resources you’ll need for evaluation, 
and developing a workplan. In Step 3, you will prepare your tools for col- 
lecting information, test them, and then collect the information you need 
to answer your evaluation questions. 

Why it's Useful Coordinators of the literacy programs where this tool was piloted 

reported that Step 3 was useful in the following ways: 

• it provided the program with a way to focus its evaluation as well as 
with tools for each group of stakeholders; 

• it helped to determine the best methods of collecting information for 
their program; and 

• a great deal of time is saved by having questions for surveys, inter- 
views, etc. already prepared. 

For an example of a program’s experience in using Step 3, please see the 
case study on page 144. 

How much time does it take? 

The average time it took the pilot sites to work through Step 3 was 
twenty-two hours. It’s important to note that collecting information for an 
evaluation can be spread out over many months, as shown in the sample 
workplan on pp. 33 - 34. 



“We submit an annual survey anyway but this one was much more 
focused and structured. I really felt that by doing this process (inter- 
. views, focus groups, etc.) we really had direction for change and 
improvement. I can’t stress enough how much I needed to know 
what I should be doing, how I should be doing it, and what steps I 
need to take to get there or start there.” 

— pilot site coordinator 
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Tips on using different methods 



Before you prepare the tools you will use to collect information, here are 
some tips on how they should be developed and used. (Please note that 
we do not include journals, portfolios, and tests of student performance in 
the following list of methods. These are invaluable tools, but they are 
ways of assessing student progress, rather than tools for program evalua- 
tion. The compiled results of student assessment are what are needed for 
program evaluation, which would require reviewing relevant documents 
or records.) 



In general 



Assure all respondents that their responses will be anonymous and that 
their names won’t appear anywhere in the information. 

For all of the suggested methods of collecting information, make sure 
that there is an explanation included of the reason why the information is 
being collected, i.e. to evaluate the program, not to evaluate the person. 

The tools and questions that you use to collect information must be reli- 
able and valid. “Reliable” means that the way you collect information 
must be accurate and consistent. For example, if different interviewers 
ask the same question of a student, they should get similar answers. If a 
person responds differently when asked the same question in a short 
period of time, you should have concerns about the reliability of the ques- 
tion. “Valid” means that the tool used to collect information actually 
measures what it is supposed to measure. For example, if you need to find 
out if students and tutors took part in developing the program’s mission, 
and the interview question asks if they are aware of the program’s mis- 
sion, the question is not valid. 

When you are developing your surveys, interview forms, and focus 
groups, use the following as guidelines: 

• arrange items so that the more personal, difficult, or potentially contro- 
versial questions come later in the document; 

• organize the items by the type of responses, i.e. group the “yes/no” 
questions together, the satisfaction questions together, etc.; and 

• as far as possible, organize the items according to topic, for example, 
put all of the questions relating to tutor training together. 

The total number of questions on a survey or in a focus group should be 
reasonable in terms of the respondents’ time and in terms of handling the 
amount of information collected. (You could use approximately 15-20 
semi-structured questions like those in Appendix B for a 30-minute inter- 
view.) If you are collecting information from a large number of people, 
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Document review 



Observation 




Setting the Compass 



you may have to limit the number of open-ended questions you ask, 
because of the time that would be needed to compile and analyze the 
information. 

When preparing forms, use a font and size that is easy to read. Don’t split 
questions between pages. Put questions in a different font or style so that 
they stand out from the instructions or the response categories. 

Finally, when you are preparing your tools to collect information, make 
sure that you develop and include questions that will provide information 
for the items you added in Appendix A under “other.” 

“Document review” is suggested as the method of collecting information 
for several standards in Appendix A. Reviewing documents involves 
looking at secondary data, in other words, information that has already 
been collected for another purpose and recorded in a text document or 
computer file. In many cases reviewing documents is sufficient for many 
information needs, and collecting information in other ways is not neces- 
sary. 

Documents might include program records, reports, letters, memo, e- 
mail, training materials, and student records. Having information docu- 
mented is not only important for evaluation, but for comparing a new 
approach to what has happened in the past. 

Documents could be reviewed by an evaluation team member, aided if 
necessary by staff. Remember to consider confidentiality issues when 
deciding on who is going to review documents; you may want to prepare 
a contract with the person that addresses the need to keep information 
confidential. 

Viewing events and actions of people is useful as a method of collecting 
information when the information required has not been previously col- 
lected and is not likely to be available through survey, interview, or focus 
group. In writing a description of what is being observed, the evaluator is 
making a judgment about the nature and quality of the event. The 
descriptions must be factual, accurate, and not cluttered by irrelevant 
details. 

Like individual interviews, observation is more time consuming than 
other methods of collecting information, and therefore is only recom- 
mended a few times in Appendix A. It is, of course, a very useful method 
of collecting information, as it yields immediate and concrete informa- 
tion. (Seeing is believing!) 
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Surveys 
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As with doing interviews (described below), you can decide on the num- 
ber of observations to make in different ways. One is to use the criteria 
described in that section, and the other is to use 30% as a rule of thumb 
(i.e. observe 30% of your student-tutor pairs), with a minimum of 10 
observations for any particular standard. 

There are different ways of doing observation, including “onlooker” and 
“participant” observation. Both are respected methods, but participant 
observation requires a great deal of training and is not recommended 
here. As an onlooker, you should refrain from being drawn into the tutor- 
ing session in the way of providing advice, demonstrating techniques, 
etc. The idea is to see the extent to which the activities taking place are 
meeting the standard. 

Record the names of the student and tutor only to keep track of who has 
and hasn’t been observed. The names won’t be included when the infor- 
mation is compiled. 

Surveys are questionnaires that are mailed out or distributed for people to 
fill out independently. For example, you may include a tutor survey in 
your newsletter, or hand it out to tutors when they come in to meet with 
their student. They should be returned in such a way that respondents can 
remain anonymous, such as having a stamped, self-addressed envelope 
included, or a box set up in the tutoring area, rather than people handing 
it to staff. 

Surveys could be used with tutors, board/council/committee members, 
and community partners. They are not the best choice for getting infor- 
mation from students because of the reading and writing involved. They 
could also be used to obtain staff views if there are enough staff in a pro- 
gram to merit a survey. 

How many surveys do you need to send out? It depends how statistically 
reliable you want your findings to be. In sending surveys out to a sample 
of the group you’re surveying, if you’re dealing with a group of 10, 40, 
75 or 150, the sample size would need to be 9, 36, 62, and 108 respec- 
tively to provide a 95% confidence level within 5% degree of accuracy. 
These sample sizes represent the number of ‘completed returns' that 
would be required. Obviously, this level of scientific rigour in research is 
not likely to be feasible in the context of a volunteer literacy program. It’s 
recommended, therefore, that if you use a survey you send it out to all of 
the members of that group, i.e. tutors, board members, etc. 
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Interviews 



Face-to-face interviews could be used with students, as could telephone 
interviews if appropriate. The interviews should be conducted by some- 
one other than the student’s own tutor or the program staff, so that the 
students being interviewed could feel free to express their views of the 
program and their tutors. This could be another tutor, a board/council/ 
committee member, an evaluation team member, or a person from a part- 
ner agency. ESL students may need a family member to interpret, or a 
bilingual interviewer, if either of these is possible. 

Interviews can also be used with board/council/committee members, 
community members, and other participants. Interviews are much more 
time consuming than other ways of collecting information, so you may 
prefer to just use them for students. 

How many people do you interview? As with surveys, the small number 
of respondents in volunteer literacy programs makes it difficult to come 
up with a meaningful number for a sample. At the same time, interview- 
ing all of the students, for example, would be too time consuming for 
many programs. One way to decide would be to develop certain criteria 
for who gets interviewed, and ensure that your criteria cover all of the 
different types of students you have in your program, such as male/ 
female, under 35/over 35, employed/unemployed, work-oriented/school- 
oriented, and so on. You would select one student for each criterion, and 
interview them. 

Another way of deciding how many people to interview is to use 30% as 
a rule of thumb. In programs with ten or fewer students, interview all of 
them. (Small programs would not be able to generalize about their ser- 
vices from interviewing one to three students.) 

Focus groups A focus group is a group of people who are selected because they are 

knowledgeable about the subject being investigated. A focus group is 
generally made up of 6 - 10 people who are similar in some important 
way, for example, a group of students, a group of tutors, etc. Focus 
groups are different from discussion groups in that they are brought 
together to focus on specific questions or topics. The questions in Appen- 
dix B can be used to guide discussion in a focus group, but the informa- 
tion generated will likely be more in depth than you would get using a 
survey or interview. 

Focus groups could be designed for any or all of the above groups, to 
confirm or add to the results of surveys or interviews. They should be 
conducted by someone who is not known by focus groups participants, 
again to allow the participants to speak freely. Such a person might be a 
member of the evaluation team or a member of the community. The per- 
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son leading the focus group should have some facilitation skills that 
would help to manage difficult participants, encourage reluctant partici- 
pants, and record information while at the same time asking questions 
and keeping the group focussed. 



It is very important to use a set of “ground rules” with focus groups. This 
can prevent difficult or awkward situations from arising during the dis- 
cussion, and helps to make expectations of group members clear. Typical 
ground rules include items such as keeping all information discussed 
confidential, treating everyone with respect, allowing all members of the 
group to participate, agreeing on what important words mean, asking for 
clarification if necessary, listening to everyone’s opinions, allowing for 
disagreement, and staying focused and on time. 



Reviewing current methods 

Depending on what evaluation practices already exist in your program, 
there may be existing methods that you can continue to use to collect 
information. 



Instructions: Review the surveys, questionnaires, focus group questions, 
or other tools for collecting information that your program is currently 
using for evaluation. Will they work for answering your selected evalua- 
tion questions? You may want to use them as they are, revise them, or 
prepare tools by using the items included in this document (see below). 



Note that if you develop your own surveys, etc., you will want to make 
sure that you can relate the collected information back to the evaluation 
questions it is meant to address. Each item on the surveys, questionnaires, 
etc. should be coded accordingly. For example, an item on your question- 
naire for tutors might ask them whether they feel adequately supported 
(evaluation question #4 in Step 2). This question number should be writ- 
ten on the survey, etc. next to the item so that information can be easily 
organized according to evaluation question as information comes in. (You 
will find references to resources on designing surveys and other data col- 
lection instruments in the Bibliography in Appendix F.) 



“This is the first time evaluation has been done by a committee, 
rather than the coordinator alone, for this program. Having more 
people involved in the process helps to ensure objectivity.” 

— pilot site coordinator 
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Reviewing relevant documents 

Instructions: Review your selections in Appendix A where “document review” is a selected method of 
collecting information. List the standard selections from “Program Operations” in the following 
chart, and as you review the relevant documents, summarize the documentation in terms of how it 
relates to the standard. For “Program Results” questions, use the worksheets included at the back of 
Appendix A (pp. 92 - 99) to record information. 



List evaluation question/ 
standard 


Summarize the documentation in relation to the evaluation 
question/standard 


example: 4.4 The program 
provides initial tutor training 
in at least the following 
areas.... 


The outline of initial tutor training shows that the training 
includes everything mentioned in the standard except for an 
introduction to literacy. 



















(Photocopy and save original ) 
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List evaluation question/ 
standard 


Summarize the documentation in relation to the evaluation 
question/standard 



























(Photocopy and save original) 
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Completing observations 

Instructions: Review your selections in Appendix A where “observation” is the selected method of 
collecting information. List the standards in the following chart. Explain to the student and tutor 
being observed that you are collecting information to see how well the literacy program is working. 
Ask them if they mind if you sit in on their session for half an hour. Do not tell them which standards 
you are making observations for. Remind them that they are not being evaluated, the program is. 



Try to be as unobtrusive as possible. Complete the following chart. 



Date 


Who 


List evaluation question/ 
standard 


Describe what is observed 


Mar. 

3/99 


Joe & 
Jim 


example: 3.12 The tutor 
uses instructional meth- 
ods that enable the stu- 
dent to apply the skills 
being learned. 


• after the tutor explained and demonstrated how 
word patterns work, the student and tutor prac- 
ticed combining different consonant sounds with 
rhyming endings 

• the student used a copy of an experience story to 
circle words with the same word pattern 



























(Photocopy and save original) 
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Date 


Who 


List evaluation question/ 
standard 


Describe what is observed 











































( Photocopy and save original) 
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Preparing surveys, interviews, and focus group questions 

In this section you will prepare the documents you need to conduct sur- 
veys, interviews, and focus groups. There is a list of “items” or questions 
that you can use to create these documents in Appendix B, as well as 
instructions to include with each tool. All you need to do is to find the 
items in Appendix B that correspond to the selections you made in 
Appendix A. 



Example: The evaluation team made these choices in reviewing Appendix A: 



Standards 


Choose at least one 
indicator unless 
otherwise noted 


Possible 
sources of 
information 


Recommended 

methods 


Date 


4.2 The program has 
clearly written and real- 
istic job descriptions for 

its tutors. w 


• existence of job 
descriptions 

PLUS 

• tutor feedback w 


• job descriD- 
tions 

• tutors 


• document 
review 

•interview, sur- 
vey, focus group 





The team wants to use a survey to get tutors’ feedback on tutor job 
descriptions. This is #4.2 in the standards, so in Appendix B the team 
looks up the items in the “Tutor” items that correspond to #4.2 (The stan- 
dard reference is shown in brackets after each question): 

“In your opinion, is the tutor job description clearly written? (4.2)” 
Yes No Don’t know 

Please explain 



This is one of two items corresponding to #4.2. Both of these items 
would be used on the tutor survey. The team would continue to find all of 
the items in Appendix B that address the standards and Program Results 
questions they have chosen, where “survey,” “interview,” or “focus 
group” is the method of collecting information. 

As noted above, make sure that you keep the reference to the evaluation 
question and standard (“4.2” in our example) with each item on the sur- 
vey, focus group, etc. so it’s easy to relate the collected information back 
to the evaluation question. 

Instructions: Using the items and instructions provided in Appendix B, 
prepare the tools you need to collect information from students, tutors, or 
other program participants, according to your selections in Appendix A. 
There is an example of a tutor survey on the following page. 
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TUTOR SURVEY (Example) 



Instructions: Please take a few moments to complete this survey, and return it to the program no 
later than April 15. (Use the stamped envelope provided, or drop it off at the program office.) You 
do not need to put your name on the survey. 

Listed below are seven questions that will help us to evaluate our program and its services. Circle 
the most appropriate answer, and include an explanation wherever possible. Thank you! 

1. Did you get useful lesson-planning information from your student’s initial assessment? (3.3) 

Yes No Don’t know 

Please explain 



2. In what ways does your student participate in defining and revisiting his or her own goals? (3.6) 
Please describe 



3. In what ways does your student participate in his/her assessments? These include assessments 
during the program and, if possible, when the student leaves the program. (3.8) 



Please describe 



4. In your opinion, is the tutor job description clearly written? (4.2) 
Yes No Don’t know 

Please explain 



5. In your opinion, is the tutor job description realistic? (4.2) 
Yes No Don’t know 

Please explain 



O 

ERIC 
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Testing the tools 



Before you begin to use the different tools you’ve developed to collect 
information, it’s important to practise with them to make sure they will 
work smoothly. You can do this by having some of your evaluation team 
members or program volunteers test the surveys, etc. on each other. In the 
case of interviews, it would be a good idea to have the same people who 
will be conducting the interviews test the tools. 



When testing the information gathering tools, have people take turns 
reading the instructions and asking questions of each other. Ask them to 
give feedback on the following: 

• Is the wording clear and unambiguous? 

• Do any of the questions make them feel uncomfortable? 

• Do they get the answers they need? (Does the tool measure what it is 
supposed to measure?) 

• Will people be able to follow the instructions? 

• Does the format make the document easy to use? 

• Are there any typographical errors or spelling mistakes? 



The items contained in Appendix B have been extensively tested with lit- 
eracy coordinators and other program participants. If you find that any of 
these items are difficult to use, please let the LCA know. 



Based on what happens when your information gathering tools are tested, 
you may need to revise your questions and format. 



One of the pilot sites asked five students to fill in a student survey as 
a means of testing it. They were asked to tell the coordinator if there 
were any points that were not clear. No one asked questions or made 
comments about the questionnaire. However, a review of the 
responses indicated several problems; students misread words and 
the results were therefore of limited use. 

Filling in a survey is an authentic reading and writing task for stu- 
dents, but this coordinator suggests using a survey with a group of 
students, and having a staff person or volunteer read and clarify the 
questions. 




^2 
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Distributing surveys, conducting interviews, and holding focus groups 

By this point in the evaluation process, the tasks, roles, timelines, and 
tools should be developed and in place for gathering the information you 
need to answer your evaluation questions. 



Information will be collected over the program year or cycle. Keeping 
your information well organized will save time and effort when it comes 
to interpreting the information and coming to conclusions about your 
program. Make sure you have clearly labelled files for information col- 
lected from students, from tutors, etc. Ensure that all information is 
dated, and the name of the person who collected the information is 
recorded in case clarification is needed. 



It is also important to keep track of your information collection activities. 
For example, with mail surveys, there should be a log of how many sur- 
veys were mailed out and how many were returned. You should also 
record how many telephone or face-to-face interviews were attempted 
and how many were completed. 



There is no real standard for response rates for any of these information 
collection methods. The acceptable level of response depends on, among 
other things, the questions asked, the type of tool used, the design of the 
tool, who is responding, and how long people have to respond. If your 
response rate is very low, consider trying to get the information using a 
different kind of method, contacting those who didn’t participate (if this 
is known), or offering incentives for participating. 



Instructions: Review your selections in Step 2 where “survey,” “inter- 
view,” or “focus group” are the selected methods of collecting informa- 
tion. Using the workplan completed in Step 2 and the information 
provided earlier in this section, collect the information required using the 
tools you have prepared. 
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Step 4 



Interpreting and Using Your Findings 



Purpose Of Step 4 In Step 3 , you used one or more methods for collecting information about 

your program. As that information was collected, you organized the 
information by putting it into file folders labelled “Student,” “Tutor,” etc. 
The next step is to summarize and interpret that information, and then 
draw conclusions that you can use for program development and account- 
ability. 

Why it's useful When this tool was piloted, coordinators reported that they found Step 4 

useful because: 

• the charts provided to organize information were helpful; 

• it helped to identify the strengths and concerns of program members; 
and 

• it generated ideas on how to present findings to stakeholders. 

For an example of a program's experience in using Step 4, please see the 
case study on page 144. 

How much time does it take? 

It took the pilot site programs an average of seven hours to complete Step 
4. 
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Compiling the information 



For each respondent type (student, tutor, etc.) and each information col- 
lection tool (interview, survey, etc.), list the responses to each item. If the 
item is a closed-ended question (yes/no, satisfaction rating, etc.), then 
tally the number of responses. If the response is to an open-ended ques- 
tion (“please explain,” “In what ways...,” etc.), list the responses, except 
if they are reasonably similar, in which case you can list it once and then 
tally the number of times it appears. 



As you are tallying responses, be sure to note the total number so that you 
can calculate averages later in the process. 



example: l\itor Survey 

4. In your opinion, is the tutor job description clearly written? (4.2) 
Rating scale: (15 responses in total) 

1 1 x “5” (very satisfied) 

• 4 x “4” 



Comments: 

• the section on activities and the program philosophy could be clearer 

• very clear (7 responses) 



5. In your opinion, is the tutor job description realistic? (4.2) 
Rating scale: (15 responses in total) 



• 9 x “3” 



• 5 x “2” 

• 1 x “1” (not at all satisfied) 



Comments: 

• it seems to imply that we work with our students in our homes! 

• much of the information seems out of date (4 comments) 

As you’re compiling information, avoid identifying people by name or by 
any other means. The anonymity you promised people while collecting 
the information needs to be maintained. 



One pilot site coordinator found that the easiest way to compile infor- 
mation was to use a blank survey form and tick off the responses beside 
each 1 - 5 scale item. She had put the tutor, student, and administration 
surveys on the computer and just typed all of the comments under each 
question. 
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Summarizing the information 



To summarize the information that you've collected, you need to summa- 
rize responses to closed questions by coming up with an average for the 
rating scale responses. You'll also need to summarize comments that were 
made, and note how many different respondents commented in a similar 
way. 



Instructions: Using the tallies that you made in “compiling information ,” 
above, calculate the average response for each rating scale question. 



example: Tutor Survey 

4. In your opinion, is the tutor job description clearly written? (4.2) 

(11 x 5) + (4x4) = 71 

71 -s- 15 responses in total = 4.7 

5. In your opinion, is the tutor job description realistic? (4.2) 

(9 x 3) + (5 x 2) + (1 x 1) = 38 

38+15 = 2.5 



Therefore, the average responses to these questions were 4.7 and 2.5 
respectively, on a scale of 1 to 5. 



Instructions: For longer descriptive responses review all of the informa- 
tion relating to a questionnaire item, focus group topic, etc. As you’re 
reviewing the information, make a note of patterns, themes, or categories 
that you start to see. Once you’ve finished reading the information 
through once, think of a code word for each of those themes or catego- 
ries, such as “support,” “access,” etc. Write a brief definition for each 
code word, such as “support from program staff,” for “ support ” and 
“ access to program, including hours of operation, location” for 
“access.” Go back to the information and write the code word beside 
related pieces of text. Then summarize the information under each code 
heading. 

Example of assigning codes, code definitions, and a summary of 
open-ended information after coding: 

7. In what ways does your student participate in his/her assessments? 
(3.8) 

A. The way that Ellen’s progress in the program is assessed seems to 
have more to do with how she uses language than measuring how much 



change 
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content 

pxvttn&tdAip, 



change 



content 




she knows about language. We ask her to tell us about what she’s learned 
and how she’s using it, rather than testing her on grammar, phonics, or 
spelling. One of Ellen’s goals that we’ve been working towards is to 
improve her reading and writing so she could help her daughter do better 
at school. Part of her assessment last month was to see how well she 
could read a report card and write a note to the teacher. Ellen sees herself 
as a partner in her learning, and that includes in her assessments. 

B. He’s asked for his opinion about what he’s learned, not just tested. 
He’s asked if anything’s changed outside the program because of what 
he’s learned. If he doesn’t think a question in the assessment interview 
has anything to do with him, we explain why it’s there. 

Code definitions 

content: assessment is related to instructional content 
change: student describes changes taking place 

partnership: the student views the assessment as an extension of the tutor- 
student relationship 

Change Content Partnership 

student describes assessment related student sees herself as 

changes in skills (2) to goals, instruction an equal in the assessment 

student can question 
content of assessment 



It’s important to simply summarize and tally information at this point, 
and not to interpret or include your views with those of your information 
sources. (Keeping track of your thoughts in a separate document is fine, 
but you want to avoid a situation where your comments could be con- 
fused with responses from students, tutors, etc.) 

As you’re working with the collected information, remember that 
although you need to summarize descriptive or qualitative information in 
order to gain an overall picture of your program, it is usually effective to 
use anecdotes and quotes, especially of students’ or tutors’ accounts of 
the program, when you report your findings to your stakeholders. 




5 



n 

l 
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Considering additional factors 



Instructions: Before using the summarized information to address standards and answer evaluation 
questions, consider any factors that contributed to or impeded the program's success during this pro- 
gram cycle. These may be things that happened outside of the program's control, as in the example 
below for “impeding factors,” but which could be important in reaching your conclusions about your 
program. 



Information needed 


Description 


contributing factors 


(Example: The regional health authority has recognized literacy 
as a priority in its prevention campaign for 1998-99, which has 
increased our credibility in the community.) 


impeding factors 


(Example: Because of the transient nature of the population , 
many students do not stay in the program more than 4-6 weeks,) 

4 



(Photocopy and save original) 
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Determining whether you are achieving the standards you selected 

This is the point in the evaluation process where you start to interpret the 
information that you’ve collected, compiled and summarized. Applying 
the information to the standards, and later to the evaluation questions, 
involves making decisions about and judging what goes on in your pro- 
gram; in other words, evaluating it. Because it is natural for people to 
bring their biases to the job of interpreting information, it is recom- 
mended that the task be shared by evaluation team members rather than 
being done by one individual. 

How will you know if your program is achieving a standard? As men- 
tioned above, evaluation involves making judgments and a certain degree 
of subjectivity. However, where 5-point rating scales are used to deter- 
mine satisfaction, frequency of an activity, etc., a rating of less than 3 
could indicate that more development is needed in that area before the 
program could be considered to be achieving that standard. Before mak- 
ing that decision, of course, you would need to look at other information 
collected during the evaluation, including comments written in response 
to open-ended questions, document reviews, etc. 

Instructions: Use the form on the next page. For each standard selected, 
list the summarized information. Identify the respondent group (student, 
tutor, etc.) Make sure you include the responses to all methods of collect- 
ing information: document review on pp. 43 - 44, observations on pp. 45 
- 46, surveys, interviews , focus groups, etc. 



Standard 


Summarized information 


Are we achieving the 
standard? 


What needs to 
happen? 


example: 4.2 The 
program has 
clearly written and 
realistic job 
descriptions for its 
tutors. 


Document review: a job 
description is on file, but is not 
current 

Tutors survey (scale of 1-5): 
average of 4.7 - clearly written 
average of 2.5 - realistic 


No 


update the 
tutor job 
description 


example: 4.6 The 
program assesses 
the effectiveness of 
its tutor training. 


Document review: the feed- 
back collected after training 
sessions is on file 


Yes 


n/a 
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Standard 


Summarized information 


Are we achieving the 
standard? 


What needs to 
happen? 




































(Photocopy and save original ) 
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Answering your evaluation questions 



Instructions: For every evaluation question that you selected in ‘‘Pro- 
gram Operations” in Step 2, list the standards that have been achieved, 
as well as those that haven’t been achieved. If there are aspects of your 
program’s context ( Step 1) or additional factors (Step 4) that affect 
whether or not a standard has been achieved, be sure to note them. For 
every evaluation question in “Program Results ” that you selected, pro- 
vide a summary of the information. 

Example: 

3. Our program delivers programming that addresses the needs of the 
individual in that: 

• the program provides each student with appropriate initial assessment; 

• each student participates in defining and revisiting his/her own goals; 

• methods used enable the student to apply the skills being used. 

We need to: 

• review our on-going and exit assessment practices and adapt them to 
the needs of our students. (Note: due to the nature of employment in 
the area, students often leave the program with little or no notice, 
which makes assessment difficult.) 

4. Our program provides quality tutoring in that: 

• we assess the effectiveness of our tutor training; 

• staff help tutors to evaluate their effectiveness and support them in 
improving their techniques if necessary. 

We need to: 

• update our tutor job description; 

• add an introduction to literacy to our initial tutor training workshop. 

5. Our program has sound administration in that: 

• we have a clearly defined relationship with our hosting authority; 

• our fiscal management includes an annual budget approved by the 
advisory committee, regular reports to our advisory committee on our 
financial status, and an annual financial review. 



7. The following services were delivered: 

• 25 students participated for a total of 1250 hours 

• 23 tutors participated for a total of 1525 hours 

• 7 other volunteers participated for a total of 75 hours 

• 2 presentations on literacy were made to community agencies 

Student satisfaction with the program includes: 

The program teaches students what they need to learn: 

• average response of 4.2 on a scale of 1 - 5 
Students can use what they learned in their daily life. 

• average response of 3.9 on a scale of 1 - 5 
The quality of the services provided was good. 

• average response of 4.6 on a scale of 1 - 5 

• The program will help students to achieve their goals. 

• average response of 4.3 on a scale of 1 - 5 
Overall, students are satisfied with this program. 

• average response of 4.6 on a scale of 1 - 5 
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Making conclusions about your program 



Now that you have compiled and summarized the information you col- 
lected during the evaluation, and used the information to look at your 
program in relation to the standards and the evaluation questions, you 
need to draw conclusions about your program. What do your evaluation 
findings say about your program as a whole? Conclusions could focus on 
such things as: 

• What are the program’s strengths? 

• Are there areas where program practice could be improved? 

• What are the results (expected and unexpected) of the program? 

• Is the program meeting its goals? 



Depending on the audiences for your evaluation, you may want to draw 
different things out in your conclusions. For example, you may wish to 
use Program Results information (see pp. 92 - 99) to emphasize the value 
of the program (in volunteer hours, etc.) for the purposes of gamering 
local support for your program. 



Your conclusions should indicate what impact your program had on its 
participants. One of the items in Program Results question #7, for exam- 
ple, has to do with whether the program will help students to achieve 
their goals. Your conclusions might then state, “Results show that 72% of 
our students were very satisfied that the program will help them to 
achieve their goals.” 



Instructions: Develop conclusions based on what you have learned about 
your program during this evaluation. Then, using these conclusions, 
decide on follow-up actions or changes to the program. Make sure these 
changes are incorporated into program planning. 



One pilot site’s experience in drawing conclusions: “We got our eval- 
uation team together again and discussed each standard and the infor- 
mation gathered. It was such a positive experience to have 
confirmation of our success in our community.” 
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Informing others 



Celebrate! 



Instructions: Looking at the audiences that you identified in Step 2, along 
with the reasons why they would need evaluation information, decide 
how you will inform them of relevant evaluation findings and any 
changes to the program resulting from the evaluation. 

Depending on your program and the needs of your stakeholders, informa- 
tion about the evaluation can be communicated: 

• through an article in the program newsletter; 

• in a written report; 

• in a presentation; and/or 

• in an informal discussion. 

Once again, in addition to the summarized information and conclusions 
that your evaluation team has come to about your program, be sure to 
include some quotes from your program participants in your evaluation 
reports. Their stories and anecdotal comments are powerful ways of com- 
municating the program’s successes. 

If a written evaluation report is prepared, it should be circulated to the 
evaluation team for feedback before being distributed. 

Finally, an essential component of program evaluation should be for par- 
ticipants to celebrate the program’s accomplishments. A very important 
result of having standards for literacy programs should be to affirm the 
excellent work currently being done in volunteer literacy programs across 
this province. The following story was offered by a literacy coordinator 
during the 1996 ALPS consultations. It’s just one example of why we 
should celebrate. 



“ ^ottifish man who had. to learn from a cfjiade 1 level 
called me on C Zhristmas morning to tell me he had read 
“‘^Cwas tf^he AJi^ht 7sefoie (Christmas” to his children, 
the fast time he had ever read to them. heif were both in 
their late teens.” 
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Appendix A Applying the standards and evaluating 

program results 

Purpose of Appendix A 

The tables in Appendix A are to be used as part of Step 2, when you 
select the standards that you are going to work with to answer the ques- 
tions you have for this evaluation. (See page 28.) It is very important that 
you follow the instructions found in Step 2 so that you know how much 
you can include in your evaluation given the time and resources you have 
available. 

There are also tables at the back of this appendix that are to be used in 
Step 2 when you are selecting items from Program Results questions #7 - 
9 (see page 29), and in Step 3 when you are collecting information to 
answer those questions. 

Why it's Useful Pilot site programs found Appendix A useful because: 

• the tables were easy to follow, and kept the focus on the particular 
standards the program is working on; 

• knowing these tables exist to be used as a follow-up to selecting stan- 
dards makes this job less daunting; 

• there is a good selection of indicators for each standard; 

• the example at the end of the appendix was very useful; and 

• it was very useful for someone with no experience in program evalua- 
tion. 



Instructions: Before using these tables, make a photocopy and save these 
pages as originals. Use your photocopy to make your selections, either 
by underlining, checking off, or highlighting your selected items. Use the 
last column to enter the date when the standard was used in program 
evaluation. (See page 100 for an example of how to use these tables.) 

Note : you may find yourself answering “yes” or “no” to the standards as 
you review them. Try to think in terms of showing support for your 
answer, as it will make the task of selecting indicators, sources of infor- 
mation, and methods of collecting information much easier. 

If you would find it easier to consider the standards in groups within each 
of the six evaluation questions, refer to Appendix D, where standards are 
grouped together under good practice statements. 
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Program Operations Questions 



l In what ways is our program community based? 



Standards 


Choose at least one 
indicator unless 
otherwise noted 


Possible 
sources of 
information 


Recommended 

methods 


Date 


1.1 The program 
assesses the literacy 
needs of its community 
at least once every three 
years. 


• presence/absence of 
needs assessment pro- 
cess 

• collaboration with 
other community groups 
and agencies to gather 
information on commu- 
nity literacy needs 

• use of information 
gathered by other groups 
and agencies 

• use of participant 
knowledge of commu- 
nity needs 

PLUS 

• frequency of assess- 
ment 


• records of 
needs assess- 
ment activities 

• correspon- 
dence with com- 
munity groups 
and agencies, 
records of 
needs assess- 
ment activities 

• records of 
needs assess- 
ment activities 

• records of 
needs assess- 
ment activities 

• records of 
needs assess- 
ment activities 


• document 
review 

• document 
review 

• document 
review 

• document 
review 

• document 
review 




1 .2 Taking into account 


• match between services 


• results of 


• document 




the program’s philoso- 


offered and community 


needs assess- 


review 




phy, vision, and available 


needs 


ment activities, 






resources, the program 




description of 






responds to identified 




services offered 






needs with appropriate 


• referral agency satisfac- 


• referral agen- 


• interview, sur- 




services. 


tion 


cies 


vey 




1 .3 The recruitment pro- 


• range of contacts made 


• reports on 


• document 




cess for board, council, 


for recruitment purposes 


recruitment 


review 




or committee members is 


• community, student, 


efforts 






designed to gain repre- 


and volunteer participa- 


• membership 


• document 




sentation from a broad 


tion on board/council/ 


list of board/ 


vlV/ V 141 A 1V1 1 t 

J-m AIT / 




range of community 


committee 


council/commit- 


review 




groups and program par- 




tee, Lit-Link 






ticipants. 
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Standards 


Choose at least one 
indicator unless 
otherwise noted 


Possible 
sources of 
information 


Recommended 

methods 


Date 


1 .4 The expectations and 
responsibilities of the 
board, council, or com- 
mittee are clearly docu- 
mented and 

communicated to mem- 
bers and prospective 
members. 


• content of recruitment 
and orientation materi- 
als for board/council/ 
committee members 

• board/council/commit- 
tee member satisfaction 

PLUS 

• existence of docu- 
mented expectations and 
responsibilities 


• minutes, corre- 
spondence, out- 
line of 
orientation 

• board/council/ 
committee 
members 

• minutes, corre- 
spondence 


• document 
review 

• interview, sur- 
vey, focus group 

• document 
review 




1 .5 The board, council, or 
committee is provided 
with the Literacy Coordi- 
nators of Alberta stan- 
dards for literacy 
programs and with the 
expectations of the 
funder(s). 


• distribution of docu- 
ments 

• content of orientation 


• minutes of 
meetings 

• outline of ori- 
entation 


• document 
review 

• document 
review 




1 .6 Working with pro- 
gram staff, the board, 
council, or committee 
establishes annual goals 
and action plans that 
support the goals of the 
literacy program. 


• staff satisfaction 

• board/council/commit- 
tee satisfaction 

• existence of a planning 
document that specifies 
board goals and activities 


• paid staff 

• board/council/ 
committee 
members 

• board/council/ 
committee 
records 


• interview, sur- 
vey, focus group 

• interview, sur- 
vey, focus group 

• document 
review 




1 .7 The program ensures 
its board, council, or 
committee receives the 
support it needs to 
accomplish its literacy- 
related goals. 


• member satisfaction 
with support received 

• existence of support 
activities 


• board/council/ 
committee 
members 

• board/annual 
reports 


• interview, sur- 
vey, focus group 

• document 
review 




1 .8 The board, council, or 
committee meets at least 
two times per year to 
focus specifically on 
matters relating to the lit- 
eracy program. 


• issues discussed at 
meetings 

PLUS 

• frequency of meetings 


• minutes of 
meetings 

• minutes of 
meetings 


• document 
review 

• document 
review 
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Standards 


Choose at least one 
indicator unless 
otherwise noted 


Possible 
sources of 
information 


Recommended 

methods 


Date 


1 .9 The program seeks 


• number of approaches 


• correspon- 


• document 




out partnerships with 


made to community 


dence with com- 


review 




community organiza- 
tions and individuals 


organizations and indi- 
viduals about partner- 


munity 

organizations 






with whom it shares 


ships or joint activities 


and individuals 






complementary goals. 


• occurrence of joint 


• minutes of 








activities 


meetings, 


• document 








reports on joint 


review 






• use of community 
resource people or activi- 
ties in the program 


activities 

• reports of pro- 
gram activities 


• document 
review 




1.10 The program acts as 


• number of calls from 


• records of 


• document 




a community resource 
for literacy development. 


community members 
requesting information 


inquiries, staff 
• board/annual 


review, inter- 
view, survey 






• number of presenta- 


reports 








tions made to commu- 
nity groups or agencies 


• library records 


• document 
review 






• use of resource library 
by community members 

* attendance at literacy 
program workshops open 
to the community 


• workshop 

attendance 

records 


• document 
review 

• document 
review 




1 .1 1 The program reviews 


• existence of a partner- 


• documents 


• document 




its community partner- 


ship review process 


related to 


review 




ships annually. 


• changes to partner- 
ships based on review 

PLUS 

• frequency of review 


reviewing part- 
nerships 

• reports, plan- 
ning documents 

• board/annual 
reports 


• document 
review 

• document 
review 
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Standards 


Choose at least one 
indicator unless 
otherwise noted 


Possible 
sources of 
information 


Recommended 

methods 


Date 


1.12 Promotional activi- 
ties include efforts to 
reach potential students 
and tutors and a broad 
range of community 
groups. 


• use of former learners’ 
and tutors’ experiences 
in creating awareness 
activities 

• use of strategies for 
reaching community 
groups and potential stu- 
dents and volunteers 


• anecdotal 
records, plan- 
ning documents 

• recruitment- 
related docu- 
ments 


• document 
review 

• document 
review 




1 .13 The program recog- 
nizes community sup- 
port, and reviews its 
methods for recognizing 
its supporters each year. 


• existence of recogni- 
tion activities 

• changes in recognition 
practices due to review 
process 

PLUS 

• frequency of review 


• board/annual 
reports 

• records of 
events, reports, 
planning docu- 
ments, minutes 

• board/annual 
report 


• document 
review 

• document 
review 

• document 
review 




other 
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2. In what ways does our program have a relevant and applied philosophy? 



Standards 


Choose at least one 
indicator unless 
otherwise noted 


Possible 
sources of 
information 


Recommended 

methods 


Date 


2.1 The program ensures 
that all aspects of pro- 
gram practice are consis- 
tent with its philosophy 
and vision. 


• match between pro- 
gram goals and philoso- 
phy 

PLUS 

• match between pro- 
gram services and opera- 
tions and philosophy 


• planning docu- 
ments, philoso- 
phy-related 
documents, 
board/council/ 
committee 
members 

• participants, 
philosophy- 
related docu- 
ments 


• document 
review, inter- 
view, survey, 
focus group 

• interview, sur- 
vey, document 
review, focus 
group, observa- 
tion 




2.2 Philosophy- and 
vision-related docu- 
ments are written in 
plain language. 


• adherence to plain lan- 
guage guidelines 

• readability of docu- 
ments 


• philosophy, 
plain language 
guidelines (e.g. 
Plain Language 
Please) 

• philosophy, 
vision-related 
documents, 
readability 
instruments 
(e.g. Fry Graph) 


• document 
review 

• document 
review, readabil- 
ity assessment 




2.3 The program’s philos- 
ophy and vision are com- 
municated to all new 
staff, students and volun- 
teers in ways that are 
easy to understand. 


• ways the philosophy 
and vision are shared 

• content of orientation 


• participants 

• orientation 
outline 


• interview, sur- 
vey, focus group 

• document 
review 




2.4 Philosophy-related 
statements reflect the lit- 
eracy needs of the com- 
munity. 


• correlation between 
philosophy, etc. and 
identified literacy needs 

• satisfaction of commu- 
nity representatives on 
the board/council/com- 
mittee with program’s 
philosophy 


• results of 
needs assess- 
ment activities, 
documents 
relating to phi- 
losophy 

• board/council/ 
committee 
members 


• document 
review 

• interview, sur- 
vey, focus group 
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Standards 


Choose at least one 
indicator unless 
otherwise noted 


Possible 
sources of 
information 


Recommended 

methods 


Date 


2.5 The program has doc- 
umented mission and 
values statements. 


• existence of mission, 
values documents 


• documents 


• document 
review 




2.6 The program offers 
staff, students, and vol- 
unteers the opportunity 
for input when develop- 
ing its vision, mission, 
and values statements. 


• participant satisfaction 
with opportunity to give 
input 

• ways in which partici- 
pants are given opportu- 
nities for input 


• participants 

• program 
development 
records and 
reports 


• interview, sur- 
vey, focus group 

• document 
review 




2.7 At least every three 
years, the program 
offers staff, students and 
volunteers the opportu- 
nity for input when 
reviewing the vision, 
mission, and values 
statements. 


• participant satisfaction 
with opportunity to give 
input 

• ways in which partici- 
pants are given opportu- 
nities for input 

PLUS 

• frequency of review 


• participants 

• program 
development 
records 

• planning meet- 
ing minutes 


• interview, sur- 
vey, focus group 

• document 
review 

• document 
review 




other 






* 
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3. In what ways do we deliver programming that addresses the needs of the 

individual? 



Standards 


Choose at least one 
indicator unless 
otherwise noted 


Possible 
sources of 
information 


Recommended 

methods 


Date 


3.1 The program provides 
each student with a con- 
fidential interview as 
soon as possible after he 
or she contacts the pro- 
gram. 


• timing and location of 
interviews 

PLUS 

• handling of participant 
information 


• participant 
records, coordi- 
nator’s agenda 

• staff, staff per- 
formance 
appraisal 


• document 
review 

• interview, 
focus group, 
document 
review 




3.2 The interview is used 
primarily to explain the 
program’s services and 
to discuss any expecta- 


• content of interview 


• outline of 
interview pro- 
cess, notes taken 
during interview 


• document 
review 




tions the student may 
have. 










3.3 The program provides 


• ways in which students 


• assessment 


• document 




each student with an ini- 
tial assessment which: 


participate 

PLUS 


records, staff 


review, inter- 
view 




• is student-centered 


• existence of student 


• assessment 


• document 




and participatory; 

• neips the student to 


goals 


records, student 
files 


review 




set realistic literacy- 










related goals; 


PLUS 




• document 
review, inter- 




• provides useful 


• usefulness of informa- 


• assessment 
records, stu- 




information for the 
student and for initial 
lesson planning; 


tion collected 
PLUS 

• existence of baseline 


dents, tutors 


view, survey, 
focus group 

• document 




• establishes a baseline 
for determining 


data 


• assessment 
records 


review 




progress. 










3.4 A staff member 
clearly explains the stu- 
dent’s basic responsibili- 
ties and asks him or her 


• content of orientation 


• orientation 
outline, notes 
taken during 
orientation 


• document 
review 




to make a commitment to 
those responsibilities. 













• 
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Standards 


Choose at least one 
indicator unless 
otherwise noted 


Possible 
sources of 
information 


Recommended 

methods 


Date 


3.5 A staff member or 
volunteer introduces the 
student to the program’s 
facilities and operations. 


• content of orientation 


• outline of ori- 
entation 


• document 
review 




3.6 Each student partici- 
pates in defining and 
revisiting his or her own 
goals. 


• ways in which students 
participate in defining 
and revisiting their goals 

• student satisfaction 
with goal-setting process 


• student 
records, tutors, 
paid staff 

• students 


• document 
review, focus 
group, observa- 
tion, survey, 
interview 

• interview, sur- 
vey, focus group 




3.7 The student’s goals 
form the focus of lesson 
planning and the basis 
for on-going and exit 
assessment. 


• correlation between 
student goals, lesson 
plans, and assessments 

• student satisfaction 
with relevance of les- 
sons and assessments 


• student records 

• students 


• document 
review 

• interview, sur- 
vey, focus group 





Setting the Compass 



page 70 



Standards 


Choose at least one 
indicator unless 
otherwise noted 


Possible 
sources of 
information 


Recommended 

methods 


Date 


3.8 The program provides 
each student with on- 
going and, if possible, 
exit assessment which: 

• is student-centered 
and participatory; 

• uses appropriate for- 
mal and/or informal 
assessment methods; 

• is used to record 
gains in literacy devel- 
opment as well as in 
related areas such as 
independence and 
self-esteem; 

• provides informa- 
tion that can be used 
by both tutors and stu- 
dents to review 
progress and make 
further lesson plans; 

• is done in such a way 
that results can be 
aggregated across stu- 
dents and used in pro- 
gram evaluation. 


• ways in which students 
participate 

• amount and quality of 
support and instruction 
on self assessment pro- 
vided to students 

PLUS 

• variety of assessment 
tools and techniques 
used 

PLUS 

• range of student com- 
petency areas assessed 

PLUS AT LEAST ONE OF: 

• relevance to students 
and tutors of informa- 
tion collected 

• use of assessment in 
developing lesson plans 

PLUS 

• use of assessment 
information in program 
evaluation 


• assessment 
records, tutors, 
staff 

• workshop out- 
lines, student 
records 

• assessment 
records 

• assessment 
records 

• students, tutors 

• lesson plans, 
tutors 

• program evalu- 
ation documents 


• document 
review, observa- 
tion, focus 
group, survey 

• document 
review 

• document 
review 

• document 
review 

• interview, sur- 
vey, focus group 

• document 
review, inter- 
view, survey, 
focus group 

• document 
review 




3.9 The program uses the 


• evidence of use of 


• student 


• document 




results of the initial inter- 


information in matching 


records, Lit- 


review 




view and assessment to 


the student with a tutor 


Link 






match the student with a 










tutor. 










3. 10 The tutor and student 


• frequency of tutoring 


• tutor logs, stu- 


• document 




meet at least once per 


sessions 


dent records, 


review 




week. 




Lit-Link 
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Standards 


Choose at least one 
indicator unless 
otherwise noted 


Possible 
sources of 
information 


Recommended 

methods 


Date 


3.11 The tutor uses 
instructional strategies 
and content that are 
appropriate for the stu- 
dent's goals, skills, and 
interests. 


• evidence of use of ini- 
tial assessment results in 
tutor planning 

• student satisfaction 
with the relevance of les- 
sons 


• tutor planning 
documents 

• students 


• document 
review 

• interview, sur- 
vey, focus group 




3.12 The tutor uses 
instructional methods 
that enable the student 
to apply the skills being 
learned. 


• tutoring session activi- 
ties 


• tutor logs, les- 
son plans, tutor- 
student pair 


• document 
review, observa- 
tion 




3.13 A student leaving the 
program is encouraged 
to return if or when his 
or her future goals 
include further literacy 
development. 


• content of exit inter- 
view 

• occurrence of invitation 
to return 


• outline of 
interview, stu- 
dent records 

• student 
records, tutor 
logs 


• document 
review 

• document 
review 




3.14 Staff members pro- 
vide follow-up during 
the first month of tutor- 
ing and continue to be 
available when the stu- 
dent has questions or 
concerns. 


• timing of follow-up 
PLUS 

• student satisfaction 
with staff availability 


• student records 

• students 


• document 
review 

• interview, stir- 
vey, focus group 




3.15 Staff contact each 
student at least once 
every three months. 


• frequency of contact 


• student records 


• document 
review 




3.16 The program helps 
students find needed sup- 
port services, both within 
and outside the program. 


• number and type of 
internal support services 
available 

• student satisfaction 
with internal support ser- 
vices 

PLUS AT LEAST ONE OF: 


• student orien- 
tation materials 

• students 

• student 
records, Lit- 
Link 

• students 


• document 
review 

• interview, sur- 
vey, focus group 

• document 
review 

• interview, sur- 
vey, focus group 




• number and type of 
external referrals offered 

• student satisfaction 
with referral information 
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Standards 


Choose at least one 
indicator unless 
otherwise noted 


Possible 
sources of 
information 


Recommended 

methods 


Date 


3.17 Current information 
on other education and 
training programs is 
available and students 
are assisted in using it. 


• presence/absence of 
current information 


• pamphlets, 

program 

descriptions 

and calendars, 

registration 

forms 


• document 
review 






• staff knowledge & uses 
of current information 

• student satisfaction 
with support received 


• staff, staff per- 
formance 
appraisals 

• students 


• interview, doc- 
ument review 

• interview, sur- 
vey, focus group 




3.18 Both staff and tutor 
give the student frequent 
praise and positive rein- 
forcement. 


• frequency of praise 
and reinforcement 

• student satisfaction 
with supportive environ- 
ment 


• staff, tutor 

• students 


• observations 

• interview, sur- 
vey, focus group 




3.19 The program 
honours each student’s 
efforts and achieve- 
ments, in an appropriate 
manner, at least once a 
year. 


• type of recognition 
used 

• student satisfaction 
with recognition 

PLUS 

• frequency of events 


• planning docu- 
ments, reports 

• students 

• board/annual 
report 


• document 
review 

• interview, sur- 
vey, focus group 

• document 
review 




other 
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4. In what u/ays are we providing quality tutoring? 



Standards 


Choose at least one 
indicator unless 
otherwise noted 


Possible 
sources of 
information 


Recommended 

methods 


Date 


4.1 In recruiting tutors, 
the program clearly out- 
lines the qualifications 
required and screens 
potential tutors to ensure 
they meet those qualifi- 
cations. 


• existence of a docu- 
ment describing desired 
qualifications 

PLUS 

• existence of screening 
process 


• recruitment 
materials 

• outline of 
screening pro- 
cess, interview 
questions 


• document 
review 

• document 
review 




4.2 The program has 
clearly written and real- 
istic job descriptions for 
its tutors. 


• existence of job 
descriptions 

PLUS 

• tutor feedback 


• job descrip- 
tions 

• tutors 


• document 
review 

•interview, sur- 
vey, focus group 




4.3 An interview is used 
primarily to discuss the 
tutor’s expectations and 
to explain the program’s 
requirements. 


• content of interview 


• outline of 
interview, notes 
taken during 
interview 


• document 
review 




4.4 The program provides 
initial tutor training in at 
least the following areas: 

• introduction to liter- 
acy 

• student-centred 
approaches, methods 
and strategies in read- 
ing, writing, and/or 
math 

• available resources 
and support 

• tutor roles and 
responsibilities. 


• content of training 
workshops 

• tutor satisfaction with 
content of initial training 


• outline of ini- 
tial training 
workshops 

• tutors 


• document 
review 

• interview, sur- 
vey, focus group 
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Standards 


Choose at least one 
indicator unless 
otherwise noted 


Possible 
sources of 
information 


Recommended 

methods 


Date 


4.5 The program assesses 
the on-going training 
needs of its tutors and 
provides training to 
meet those needs. 


• use of process to assess 
needs 

• match between requests 
for training and work- 
shops offered 

• tutor satisfaction with 
training provided 


• tools used for 
assessing needs 

• results of 
assessments, 
outlines and 
records of work- 
shops 

• tutors 


• document 
review 

• document 
review 

• interview, sur- 
vey, focus group 




4.6 The program assesses 
the effectiveness of its 
tutor training. 


• presence or absence of 
training evaluation/feed- 
back opportunities 

• changes in training due 
to assessing effectiveness 


• records of pro- 
gram activities, 
evaluation/feed- 
back instru- 
ments 

• workshop out- 
lines 


• document 
review 

• document 
review 




4.7 Staff maintain contact 
with tutors who are not 
yet matched with a stu- 
dent. 


• occurrence of contacts 


• tutor records, 
Lit-Link 


• document 
review 




4.8 Staff members pro- 
vide follow-up during 
the first month of tutor- 
ing and continue to be 
available when the tutor 
has questions or con- 
cerns. 


• timing of follow-up 
PLUS 

• tutor satisfaction with 
staff availability 


• tutor records, 
Lit-Link 

• tutors 


• document 
review 

• interview, sur- 
vey, focus group 




4.9 Staff contact each 
tutor at least once every 
3 months. 


• frequency of contact 


• tutor records, 
Lit-Link 


• document 
review 




4.10 Staff help tutors with 
resource materials as 
needed. 


• tutor satisfaction 


• tutors 


• interview, sur- 
vey, focus group 




4.1 1 Staff help tutors 
evaluate the effectiveness 
of their tutoring and sup- 
port them in improving 
their techniques as nec- 
essary. 


• tutor satisfaction 

• staff satisfaction with 
opportunities and ability 
to assist tutors in improv- 
ing their effectiveness 


• tutors 

• paid staff 


• interview, sur- 
vey, focus group 

• interview, sur- 
vey, focus group 
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Standards 


Choose at least one 
indicator unless 
otherwise noted 


Possible 
sources of 
information 


Recommended 

methods 


Date 


4.12 The program pro- 
vides opportunities for 
tutors to network. 


• tutor satisfaction 

• occurrence of network- 
ing opportunities 


• tutors 

• board/annual 
reports 


• interview, sur- 
vey, focus group 

• document 
review 




4.13 Staff informally 
acknowledge the efforts 
of tutors on a regular 
basis by showing them 
that they are valued and 
appreciated. 


• tutor satisfaction 


• tutors 


• interview, sur- 
vey, focus group 




4.14 At least once a year, 
the program explicitly 
recognizes the contribu- 
tions of its tutors. 


• tutor satisfaction 

• existence of recogni- 
tion activities 

PLUS 

• frequency of recogni- 
tion activities 


• tutors 

• board/annual 
reports 

• board/annual 
reports 


• interview, sur- 
vey, focus group 

• document 
review 

• document 
review 




other 
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5. In u/hat u/ays do we have sound program administration? 



Standards 


Choose at least one 
indicator unless 
otherwise noted 


Possible 
sources of 
information 


Recommended 

methods 


Date 


5.1 Where a program has 
a hosting authority, the 
relationship between it 
and the literacy program 
is clearly defined and 
documented. 


• existence of document 
defining relationship 


• organizational 
documents, 
statement of 
governance, ori- 
entation materi- 
als 


• document 
review 




5.2 Lines of authority 
within the program are 
clearly defined and docu- 
mented. 


• existence of document 
describing lines of 
authority 


• organizational 
chart, terms of 
reference, orien- 
tation materials 


• document 
review 




5.3 Responsibility for all 
aspects of the program 
is clearly defined and 
documented. 


• existence of docu- 
ments describing respon- 
sibilities 

• content of orientation 
processes for participants 


• job descrip- 
tions 

• orientation 
outline 


• document 
review 

• document 
review 




5.4 The program follows 
the funding guidelines 
provided by its funder(s). 


• response by funders to 
funding reports 

• response to grant appli- 
cations 


• correspon- 
dence from 
funder(s) 

• correspon- 
dence from 
funder(s) 


• document 
review 

• document 
review 
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Standards 


Choose at least one 
indicator unless 
otherwise noted 


Possible 
sources of 
information 


Recommended 

methods 


Date 


5.5 The program's fiscal 
management includes, at 
minimum, 

• an annual budget that 
is approved by the 
board, advisory coun- 
cil, or appropriate 
body, 

• regular reports on the 
program’s financial 
status to the board or 
advisory council, and 
to the coordinator if 
he/she does not man- 
age the budget, and . 

• an annual review or 
audit. 


• board/council/commit- 
tee satisfaction with fis- 
cal management 

PLUS 

• existence of annual 
budget, financial report- 
ing process, and annual 
review or audit 


• board/council/ 
committee 
members, trea- 
surer’s reports 

• financial 
records, propos- 
als, minutes, 
reviewers’/audi- 
tor’s reports 


• interview, sur- 
vey, focus 
group, docu- 
ment review 

• document 
review 




5.6 The broad qualifica- 
tions looked for in paid 
staff, depending on the 
position they are being 
hired for, include the fol- 
lowing: adult education 
or training, teaching 
reading and writing, 
instructional planning, 
program planning and 
administration, volun- 
teer management. 


• staff qualifications 


• staff resumes, 
interview ques- 
tions and 
responses . 


• document 
review 




5.7 The program clearly 
outlines the specific 
qualifications required 
by its paid staff. 


• nature of information 
provided to potential 
applicants 


•job ads, terms 
of reference 


• document 
review 




5.8 The program uses 
comprehensive, clearly 
written, and realistic job 
descriptions for all paid 
staff. 


• existence of job 
descriptions 

PLUS 

• staff feedback 


• job descrip- 
tions 

• paid staff, staff 
performance 
appraisals 


• document 
review 

• interview, sur- 
vey, focus 
group, docu- 
ment review 
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Standards 


Choose at least one 
indicator unless 
otherwise noted 


Possible 
sources of 
information 


Recommended 

methods 


Date 


5.9 The program’s 
recruitment strategy 
includes using clearly 
outlined qualifications 
and personal interviews 
to screen potential volun- 
teer staff. 


• existence of docu- 
mented qualifications 

PLUS 

• existence of screening 
process 


• recruitment 
materials 

• outline of 
screening pro- 
cess, interview 
questions 


• document 
review 

• document 
review 




5.10 The program uses 
current, clearly written, 
and realistic job descrip- 
tions for volunteer staff 
positions. 


• existence of job 
descriptions 

PLUS 

• volunteer feedback 


• job descrip- 
tions 

• evaluation 
records, volun- 
teer staff 


• document 
review 

• document 
review, inter- 
view, survey 




5.11 The program pro- 
vides all new staff with 
an orientation to the pro- 
gram's policies, opera- 
tions, and context. 


• content of orientation 


• record of pro- 
gram activities, 
reports, outline 
of orientation 


• document 
review 




5.12 The program identi- 
fies the on-going profes- 
sional development 
requirements of staff. 


• existence of process to 
identify training needs 


• documents 
related to 
assessing train- 
ing needs, 
assessment 
instruments 


• document 
review 
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Standards 


Choose at least one 
indicator unless 
otherwise noted 


Possible 
sources of 
information 


Recommended 

methods 


Date 


5.13 The program ensures 
that paid staff have 
access to relevant initial 
and on-going training. 
Depending on their roles 
in the program, staff 
training includes the fol- 
lowing areas: 

• teaching reading, 
writing, and math 

• volunteer manage- 
ment 

• assessment tech- 
niques and tools 

• program and office 
management 

• building community 
partnerships 

• current literacy 
trends and research. 


• number and content of 
“in-house” training 
activities 

• number and content of 
external professional 
development opportuni- 
ties 

• staff satisfaction with 
access to relevant train- 
ing opportunities 


• outline of 
training events, 
board/annual 
reports 

• conference 
programmes, 
board/annual 
reports 

• paid staff, staff 
performance 
appraisals 


• document 
review 

• document 
review 

• interview, 
focus group, 
document 
review 




5.14 The program ensures 
that volunteer staff have 
access to initial and on- 
going training in areas 
relevant to their duties. 


• number and content of 
workshops offered 

• volunteer staff satisfac- 
tion 


• board/annual 
report, outline 
of training 
events 

• volunteer staff 


• document 
review 

• interview, sur- 
vey, focus group 




5.15 The program moni- 
tors the effectiveness of 
training for paid and vol- 
unteer staff. 


• presence or absence of 
training evaluation/feed- 
back opportunities 

• changes in training due 
to monitoring effective- 
ness 


• records of pro- 
gram activities, 
evaluation/feed- 
back instru- 
ments 

• workshop out- 
lines 


• document 
review 

• document 
review 
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Standards 


Choose at least one 
indicator unless 
otherwise noted 


Possible 
sources of 
information 


Recommended 

methods 


Date 


5 . 16 Through its links 
with other literacy orga- 
nizations at regional and 
provincial levels, the 
program provides net- 
working opportunities 
for its paid staff at least 
twice a year. 


• staff satisfaction 
PLUS 

• frequency of network- 
ing opportunities 


• board/annual 
report 

•paid staff, staff 

performance 

appraisal 


• document 
review 

• interview, sur- 
vey, focus 
group, docu- 
ment review 




5.17 The program's 
expectations of volunteer 
staff are clearly defined, 
documented, and com- 
municated. 


• content of initial inter- 
view, orientation 

• volunteer satisfaction 
PLUS 

• existence of docu- 
mented expectations 


• interview 
notes, orienta- 
tion outline 

• volunteer staff 

• volunteer pol- 
icy manual, vol- 
unteer hand- 
outs 


• document 
review 

• interview, sur- 
vey, focus group 

• document 
review 




5.18 The program pro- 
vides volunteer staff with 
the resources they need 
to complete their tasks. 


• volunteer satisfaction 


• volunteer staff 


• interview, sur- 
vey, focus group 




5.19 The program has a 
means of keeping all vol- 
unteers informed. 


• availability of informa- 
tion 


• newsletters, 
bulletin boards, 
records of social 
events 


• document 
review 




5.20 The program has 
practices in place to pro- 
tect the confidential 
nature of participant 
records. 


• existence of policy on 
confidentiality 

PLUS 

• content of training and 
orientations 


• policy-related 
documents 

• training, orien- 
tation outlines 


• document 
review 

• document 
review 




5.21 The program clearly 
communicates to volun- 
teers, students, and paid 
staff the program’s 
expectations regarding 
confidentiality for all 
program participants. 


• content of training and 
orientations 

• clarity of expectations 


• training, orien- 
tation outlines 

• participants 


• document 
review 

• survey, inter- 
view, focus 
group 
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Standards 


Choose at least one 
indicator unless 
otherwise noted 


Possible 
sources of 
information 


Recommended 

methods 


Date 


5.22 The program’s plans 
for recruiting students 
and volunteers take into 
consideration the space 
and resources available 
to the program. 


• correlation between 
program resources and 
recruitment plans 


• paid staff, staff 

performance 

appraisal 


• interview, 
focus group, 
document 
review 




5.23 The coordinator 
organizes staff time so 
that both the participants' 
needs and the program's 
administrative demands 
are met within the time 
available. 


• staff satisfaction with 
time allocations 

• board/council/commit- 
tee satisfaction 


• staff, staff per- 
formance 
appraisal 

• board/council/ 
committee 
members 


• interview, sur- 
vey, focus 
group, docu- 
ment review 

• interview, sur- 
vey, focus group 




5.24 The program uses a 
sufficient amount of its 
funding to ensure fair 
and equitable staff 
wages. 


• correlation between 
portion of budget spent 
on staff wages and rec- 
ommended amounts 


• LCA recom- 
mendations for 
salary, financial 
records 


• document 
review 




5.25 Staff informally 
acknowledge the efforts 
of volunteers on a regu- 
lar basis by showing 
them that they are valued 
and appreciated. 


• volunteer satisfaction 


• volunteers 
excluding 
tutors (covered 
in other section) 


• interview, sur- 
vey, focus group 




5.26 At least once a year, 
the program explicitly 
recognizes the contribu- 
tions of its volunteers. 


• volunteer satisfaction 

• existence of recogni- 
tion activities 

PLUS 

• frequency of recogni- 
tion activities 


• volunteers 
excluding 
tutors (covered 
in other section) 

• board/annual 
reports 

• board/annual 
report 


• interview, sur- 
vey, focus group 

• document 
review 

• document 
review 
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Standards 


Choose at least one 
indicator unless 
otherwise noted 


Possible 
sources of 
information 


Recommended 

methods 


Date 


5.27 The program has 
policies and procedures 
on file that apply to paid 
staff, volunteers, and stu- 
dents, and reviews them 
annually. 


• existence of policies 
and procedures 

PLUS 

• frequency of review 


• staff policies 
and procedures 
document, vol- 
unteer policies 
and procedures 
document, stu- 
dent policies 
and procedures 
document 

• board/annual 
reports 


• document 
review 

• document 
review 




5.28 The program keeps 
up-to-date, relevant par- 
ticipant records. 


• staff satisfaction with 
participant records 

• existence of records 


• paid staff 

• volunteer 
application 
forms, volun- 
teer interview 
notes and 
screening 
reports, atten- 
dance records 
for Tutor train- 
ing, tutor logs, 
student registra- 
tion forms and 
initial assess- 
ment results, 
staff applica- 
tions/resumes, 
staff interview 
notes, staff per- 
sonnel files, Lit- 
Link 


• interview, sur- 
vey, focus group 

• document 
review 




5.29 Tutors submit 
records of the tutor-stu- 
dent pairs' activities as 
required. 


• absence or presence of 
tutor-student activity 
records 


• tutor logs 


• document 
review 
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Standards 


Choose at least one 
indicator unless 
otherwise noted 


Possible 
sources of 
information 


Recommended 

methods 


Date 


5.30 Assessment results 
are recorded in a way 
that demonstrates the 
student’s progress 
toward his or her goals 
and the learning out- 
comes achieved. 


• articulation of progress 
toward goals 

PLUS 


•student inter- 
view and/or ini- 
tial assessment 
results, tutor 
logs, on-going 
assessment 
results 


• document 
review 






• articulation of learning 
outcomes achieved 


• student 
records, on- 
going assess- 
ment results 


• document 
review 




5.31 The program keeps 
accurate, current finan- 
cial records on file. 


• existence of current 
financial records 

PLUS 

• accuracy of records 


• financial 
records 

• financial 
records, review- 
ers’/auditor’s 
report 


• document 
review 

• document 
review 




5.32 Program staff pre- 
pare reports as needed 
for the purposes of 
being accountable and 
of informing the commu- 
nity. 


• existence of reports 


• annual 
reports, board 
reports, news 
releases, news- 
letters, minutes 
of public meet- 
ings 


• document 
review 




5.33 The program has a 
comprehensive, annu- 
ally reviewed records 
management plan that 
outlines 

• which records to 
keep and for how long, 


• existence of records 
management plan 

PLUS 

• frequency of review 


• records man- 
agement plan 

• board/annual 
report, record 
of program 
activities 


• document 
review 

• document 
review 




• who has access to 
which records, and 










• proper disposal of 
records. 
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Standards 


Choose at least one 
indicator unless 
otherwise noted 


Possible 
sources of 
information 


Recommended 

methods 


Date 


5.34 The program pro- 
vides enough resource 
materials for every tutor- 
student pair, and makes 
the materials available 
to all participants. 


• tutor and student satis- 
faction with number of 
resource materials 

PLUS 

• satisfaction of partici- 
pants with availability of 
resources 


• tutors and stu- 
dents 

• participants 


• interview, sur- 
vey, focus group 

• interview, sur- 
vey, focus group 




5.35 Resource materials 
are relevant to the experi- 
ence, background, skill 
levels, and interests of 
students and tutors. 


• student and tutor satis- 
faction 

• availability of a range 
of adult-oriented, cultur- 
ally diverse literacy 
materials and tutor 
resources 


• students and 
tutors 

• library inven- 
tory, Lit-Link 


• interview, sur- 
vey, focus group 

• document 
review 




5. 36 The program reviews 
its resource materials 
annually to ensure that 
they meet the partici- 
pants' needs. 


• correlation between 
resource materials and 
participant needs 

PLUS 

• frequency of review 


• library inven- 
tory, student les- 
son plans, 
initial assess- 
ments, Lit-Link 

* documents 
related to 
reviewing 
resources 


• document 
review 

• document 
review 




5.37 The program has 
access to facilities that 
are safe and inviting for 
all participants, and that 
are adequate both for 
administrative tasks and 
for working with partici- 
pants. 


• participant satisfaction 
with facilities 


• participants 


• interview, sur- 
vey, focus 
group, docu- 
ment review 




5.38 The program ensures 
that staff have access to 
the office and instruc- 
tional equipment neces- 
sary to carry out their 
responsibilities. 


• staff satisfaction 

• amount and type of 
equipment available 


• paid staff, staff 
performance 
appraisal 

• inventory of 
equipment 


• interview, 
focus group, 
document 
review 

• document 
review 
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Choose at least one 
indicator unless 
otherwise noted 
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sources of 
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methods 


Date 


other 
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6. In what ways is our program effectively planned and evaluated? 



Standards 


Choose at least one 
indicator unless 
otherwise noted 


Possible 
sources of 
information 


Recommended 

methods 


Date 


6.1 The program has doc- 
umented plans for the 
following program 
areas, and reviews those 
plans over a three-year 
cycle: 

• assessment of com- 
munity needs 

• community partner- 
ships 

• program promotion 
and public relations 

• initial, on-going, 
and exit assessment of 
students 

• student goal setting 

• student support 

• tutor/voiunteer 
recruitment, training, 
and support 

• staff training and 
support 

• program evaluation 


• existence of plans 
PLUS 

• frequency of review 
process 


• program plan- 
ning documents 

• minutes of 
meetings, 
records of pro- 
gram activities 


• document 
review 

• document 
review 




6.2 The program devel- 
ops a long-range plan for 
future growth and direc- 
tion and reviews that 
plan at least every three 
years. 


• existence of long- 
range plan 

PLUS 

• frequency of review 


• planning docu- 
ments 

• minutes, 
records of pro- 
gram activities 


• document 
review 

• document 
review 
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Standards 


Choose at least one 
indicator unless 
otherwise noted 


Possible 
sources of 
information 


Recommended 

methods 


Date 


6.3 The program is 
informed of and incorpo- 
rates relevant develop- 
ments in literacy practice 
into its program plan- 
ning. 


• awareness of promising 
practice (e.g. STAPLE) 

• participation in profes- 
sional development 
opportunities 

• availability of relevant 
journals and new publi- 
cations 

PLUS 

• application of learning 


• paid staff, staff 
performance 
appraisal 

• board/annual 
reports, perfor- 
mance apprais- 
als 

• library inven- 
tory, records of 
loan material, 
Lit-Link 

• board/annual 
reports, staff, 
performance 
appraisal 


• interview, 
focus group, 
document 
review 

• document 
review 

• document 
review 

• document 
review, inter- 
view, focus 
group 




6.4 The program uses the 
results of program evalu- 
ation to set goals for the 
next year. 


• integration of evalua- 
tion results in planning 
process 

• number of planned pro- 
gram changes related to 
evaluation results 


• minutes of 
planning meet- 
ings 

• minutes, plan- 

nino-f^latpH 

documents 


• document 
review 

• document 
review 




6.5 Program evaluation 
tools are compatible with 
the program's philoso- 
phy, goals, and delivery 
methods. 


• compatibility between 
evaluation tools and pro- 
gram philosophy, etc. 


• evaluation 
tools, philoso- 
phy-related doc- 
uments, paid 
staff, board/ 
council/commit- 
tee members 


• document 
review, inter- 
view, survey, 
focus group 




6.6 The program uses as 
its key indicator of effec- 
tiveness the progress 
that students make 
toward their goals. 


• content of reports to 
boards, funders, commu- 
nity 

• content of public rela- 
tions items 


• reports 

• media 
releases, arti- 
cles, brochures 


• document 
review 

• document 
review 
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Standards 


Choose at least one 


Possible 


Recommended 


Date 




indicator unless 


sources of 


methods 






otherwise noted 


information 






6.7 The program uses a 


• use of measures that 


• evaluation- 


• document 




variety of measures, both 


collect information in 


related docu- 


review 




quantitative and qualita- 


narrative form (e.g. how 


ments, reports, 






tive, throughout the pro- 


do students feel they 


Lit-Link 






gram cycle, to document 


benefit from the pro- 








its success. 


gram) 










PLUS 


• evaluation- 








• use of measures that 


related docu- 


• document 






collect information in 
numerical form (e.g. how 
many students felt they 
were progressing toward 


ments, reports 


review 






• minutes of 




their goals) 


evaluation meet- 


• document 






PLUS 

• timing of evaluation 


ings, records of 
program activi- 
ties 


review 




activities 


6.8 Program evaluation 


• staff participation in 


• performance 


• document 




includes annual perfor- 


performance reviews 


reviews, person- 


review 




mance reviews and self- 


and self-evaluation 


nel records 






pvalnatinn r»f naiH staff 




• performance 






uaiua.u*. r — * 


PLUS 


• document 






• frequency of reviews 
and self-evaluations 


reviews, person- 
nel records 


review 




6.9 Program planning 


• participant satisfaction 


• participants 


• interview, sur- 




includes giving partici- 


with opportunity to give 




vey, focus group 




pants the opportunity to 


input 


• program plan- 
ning records and 
reports 






take part in setting pro- 
gram goals and objec- 
tives. 


• ways in which partici- 
pants are given opportu- 
nities for input 


• document 
review 




6.10 The program’s eval- 


• participant satisfaction 


• participants 


• interview, sur- 




uation strategies include 


with opportunity to take 




vey, focus group 




giving participants the 
opportunity to take part 
in evaluating the pro- 
gram’s effectiveness. 


part in evaluation 

• ways in which partici- 
pants are given opportu- 
nities for input 


• program evalu- 
ation records 
and reports 


• document 
review 





Setting the Compass 



page 89 



Standards 


Choose at least one 
indicator unless 
otherwise noted 


Possible 
sources of 
information 


Recommended 

methods 


Date 


6.1 1 The program encour- 
ages participants to give 
feedback on the pro- 
gram at any time. 


• existence of opportuni- 
ties for feedback 


• participants 


• interview, sur- 
vey, focus group 




other 
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Program Results Questions 

Instructions: The tables on the following pages are to be used in Step 2 
when you are selecting Program Results questions to include in your 
evaluation (#7 - 9 on page 29), and in Step 3 when you are collecting 
information to answer these questions. As explained on page 13, certain 
pages in this document replace the Participation Summary sent out by 
Community Programs of Advanced Education and Career Development . 
Along with pp. 14 - 15, use the following five pages (#7) as “pull-out” 
sheets to send to Community Programs as well as part of this process of 
looking at program results. (Be sure to carry relevant information for- 
ward into the “Student-Tutor Participation Summary” on page 96, and to 
save a copy of the completed “pull-out” sheets for your records .) 



Question #7 in the following chart asks for very specific information 
about the adult tutoring component of your program. If you offer other 
services, you will find space at the end of #7 to include them. 



(Submit pp. 14 - 15 and pp. 92 - 96 to Community Programs.) 



Program cycle being evaluated. 

(e.g. Sept. 1/99 - June 15/00 or Apr. 1/99 - Mar. 30/00) 

Program name: 



7. What services were delivered during this program cycle? 



o 

ERIC 



Information needed 


Description (use only numbers) 


Source of 
information 


Method 


number of inquiries from 
potential students 




student 
records * 


document 

review 


number of adult students who 
received tutoring 




student 
records * 


document 

review 


gender ** 

number of male students 
number of female students 


| 

. - ^ 

|| 1 1 


student 
records * 


document 

review 






previous schooling in Canada 
y# of students) ** 

no schooling 

Grade 1 - 3 

Grade 4-6 

Grade 7-9 

Grade 10+ 

special education 


|iill 

. ..... . 

fjh ■: 

jfllfH 


student 
records * 


document 

review 














previous schooling outside 
Canada ** 
no schooling 
1-3 years 
4-6 years 
7-9 years 
10+ years 




student 
records * 


document 

review 













* all information regarding services delivered can be documented in Lit-Link and automatically calculated for year-end 
participation summaries 

** total must be the same as the total number of adult students who received tutoring 
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Information needed 


Description (use only numbers) 


Source of 
information 


Method 


students’ ages ** 

19 years or younger 

20 - 35 years 
36 - 55 years 
56+ years 
unknown 




student 
records * 


document 

review 












students’ primary reason for 
joining program ** 

education 

employment 

personal 


■ 


student 
records * 


document 

review 








students’ occupation ** 
employed 
unemployed 
self-employed 
homemaker 
student 

inmpfp 

retired 

other 






student 
records * 


document 

review 


















number of students with 
special characteristics 

physical handicap(s) 
mental handicap 
diagnosed learning disability 


, 


|f||!| | f| 


student 
records * 


document 

review 








number of students with 
English as a Second Language 

fluent (working on reading 
& writing) 

not fluent (working on 
listening and speaking) 




student 
records * 


document 

review 







** total must be the same as the total number of adult students who received tutoring 



ERIC 
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Information needed 



Source of 
information 



reasons for students leaving 
program 

obtained or changed job 

enrolled in educational pro- 
gram 

moved 

time or family commitments 

program not appropriate 

lack of motivation 

asked to leave 

completed 

other 



Description (use only numbers) 




number of students referred to 
another educational program 

after tutoring 
without tutoring 



total hours of student participa- 
tion 

being tutored 
homework (if available) 
other (if available) 









student satisfaction: 

The program teaches students 
what they need to learn. 

Students can use what they 
learned in their daily life. 

The quality of the services 
provided was good. 

The program will help stu- 
dents to achieve their goals. 

Overall, students are satis- 
fied with this program. 













(give average response, e.g. 3.5 
out of 5) 



student 
records * 



student 
records * 



tutor logs, 
student 



records * 



students 



Method 



document 

review 



document 

review 



document 

review 



interview, 

survey, 

focus group 



O 

ERIC 
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Information needed 


Description (use only numbers) 


Source of 
information 


Method 


number of inquiries from 
potential tutors 




program 
records * 


document 

review 


number of trained tutors (A) 




workshop 
records * 


document 

review 


number of tutors who tutored 
(matched) 




tutor 
records * 


document 

review 


gender (matched tutors) *** 
number of male tutors 
number of female tutors 


||j §§|f 1 Htf 1 1 sti 


tutor 
records * 


document 

review 






education (matched tutors)*** 

Grade 9 or less 

Grade 10-12 

teacher training 

college or university (other 
than teacher training) 


• : % ' ■ yf • •• yf •O! 


tutor 
records * 


document 

review 










occupation (matched tutors)*** 
employed 
unemployed 

SCII-CiTlpiwy cu 

homemaker 

student 

retired 

other 




tutor 
records * 


document 

review 
















total hours of tutor 
participation (B) 

training 

tutoring 

prep, time if available 


j % 


workshop 

records, 

tutor logs * 


document 

review 










*** total must be the same as the total number of matched tutors 
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Information needed 


Description (use only numbers) 


Source of Method 

information 


number of board/council/com- 
mittee members (C) 




minutes, document 

reports * review 


total hours of participation by 
board/council/committee 
members (D) 




minutes, document 

activity review 

records * 


number of volunteers other 
than tutors and board/council/ 
committee members (E) 




activity document 

records * review 


total hours of participation by 
other volunteers (include train- 
ing) (F) 




program document 

records * review 


total number of volunteers 
(A+C+E) 




(calculate) * 


total number of volunteer 
hours (B+D+F) 




(calculate) * 


Student-I\itor 
Participation Summary 




III fj 


number of adult students 
who were tutored (p. 92) (G) 




1 I ilflll 


number of tutors who 
tutored (p. 95) (H) 






Total number of students and 
tutors (G+H) 






number of hours students 
were tutored (p. 94) (I) 




f 


number of hours tutors 
tutored (p. 95) (J) 




li 


Total number of tutoring hours 

(I+J) 




n hh im 




(End of Participation Summary information. See page 13 for instructions.) 
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7. What services were delivered during this program cycle? (con t.) 

describe other services and activities (e.g. family literacy projects, students studying independently, 
answering requests for information, research projects, community presentations) 
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8. What luere the costs of delivering the program? 



Information needed 


Description 


Source of 
information 


Method 


actual budget expenditures 




budget 


document 

review 


dollar value of volunteer con- 
tribution (tutor hours x $15 + 
other volunteer hours x $10 = 
total) 




(calculate) 




estimated dollar value of other 
contributions (e.g. facilities 
$300 x 12 months = $3600) 
(Check #3 on page 18) 




(calculate) 




other/comments (e.g. note 
whether project funding also 
supports volunteer tutor pro- 
gram) 








Total cost of program delivery 




(calculate) 
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9. What is the impact of the program on its participants? 

(Note: information on student satisfaction is required by Community Programs, and therefore appears 
as part of #7 on previous pages.) 



Information needed 


Description 


Source of 
information 


Method 


benefits to or changes observed 
in participants’ knowledge, 
skills, attitudes, values, behav- 
ior, or status during the pro- 
gram 


(give summary of benefits, 
changes) 


assessment 

records 

(e.g. 

Progress 

Profile), 

case studies, 

Lit-Link 


document 

review 


tutors’ overall satisfaction with 
the program 


/ Atn/i /-» -i rt s> i*/)Pn/»iir/j /? rr ? ^ 

\gt,VZ, UKCf 1 

out of 5) 


tutors 


interview, 

survey 


other 














Setting the Compass 



102 



page 99 



Example of how to use the Program Operations tables 



Standards 


Choose at least one 
indicator unless 
otherwise noted 


Possible 
sources of 
information 


Recommended 

methods 


Date 


4.2 The program has 
clearly written and real- 
istic job descriptions for 

its tutors. w 


• existence of job 

descriptions w 

PLUS 

• tutor feedback w 


• job descrip- 
tions 

• tutors 


• document 
review 

•interview, sur- 
vev. focus group 


Moo. 

‘98 


4.3 An interview is used 
primarily to discuss the 
tutor’s expectations and 
to explain the program’s 
requirements. 


• content of interview 


• outline of 
interview, notes 
taken during 
interview 


• document 
review 




4.4 The program provides 
initial tutor training in at 
least the following areas: 

• introduction to liter- 
acy 


• content of training 
workshops 

• tutor satisfaction with 
content of initial training 


• outline of ini- 
tial training 
workshops 

• tutors 


• document 
review 

• interview, sur- 
vey, focus group 


JVmu. 

‘98 


• student-centred 
approaches, methods 
and strategies in read- 
ing, writing, and/or 
math 










• available resources 
and support 

• tutor roles and 
responsibili- 
ties. 










other 5xdm& are given a 
JxUor Manual that out - 
tine& fta&ic program and 
tutoring information. 


• content of tutor orienta- 
tion 


* outline of ori- 
entation 


• document 
Heuieui 





If one of the evaluation questions that a program chooses is “In what ways are we providing quality 
tutoring?”, the standards above might be seen as providing useful information. For each standard 
selected, indicators, sources of information, and methods are chosen based on the resources available. 
In this example, the program will review two documents and have one question for a tutor survey. 
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Appendix B 



Sample items for surveys , interviews, and 
focus groups 



Purpose of Appendix B 

The items or questions in this appendix are intended to be used as part of 
Step 3, when you develop the survey, interview, and/or focus group forms 
that you will use to collect information from your participants. The first 
part of the appendix lists sample instructions that you can use and adapt 
for your tools, and then items are listed for tutors, students, board/coun- 
cil/committee members, volunteer staff, and paid staff. 

Remember to select the items that match the standards, indicators, 
sources of information, and methods that you chose in Step 3. There are 
also items listed for the Program Results questions in Step 3. These lists 
are not intended to be used in their entirety as questionnaires! 

Finally, in the case of using these items for interviews or focus groups, 
research has shown the importance of asking questions exactly as worded 
on the questionnaire. Inadvertent or even very slight word changes can 
change the response obtained. If questions are rephrased or response cat- 
egories changed by the interviewer, the responses can't accurately be 
combined with the responses obtained by interviewers who used the 
exact wording. If a person docsn t understand the question, repeat it for 
them, and if they still don't understand it, go on to the next question. 

Why it's useful When this process was piloted, coordinators found this appendix useful 

because: 

• it is especially useful for programs with little or no experience in pro- 
gram evaluation; 

• a tremendous amount of time is saved by not having to create and edit 
each question; and 

• it helped the coordinator to think about all of the interest groups 
affected by her program. 



Tip from a pilot site: depending on which items are used in a survey, 
interview, etc., it might be usefiil to pull definitions from the glossary 
and include them with the instructions. One person may have a differ- 
ent understanding of “mission,” “philosophy,” etc. than another person 
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Instructions: For each survey, interview, or focus group form that you are developing, include a set of 
instructions and the items that correspond to your selections in Step 3. See page 48 for an example of 
a survey. 



(Survey Instructions) 

Instructions: Please take a few moments to complete this survey, and return it to the program no 
later than (insert date). (Use the stamped envelope provided, or drop it off at the program office.) 
You do not need to put your name on the survey. 

Listed below are (insert number) questions that will help us to evaluate our program and its ser- 
vices. Circle the most appropriate answer, and include an explanation wherever possible. Thank 
you! 



(Focus Group Instructions) 

Thank you all for coming to this meeting. Your participation will help us to understand what our 
program’s strengths are, where we need to improve, and what changes we need to make, if any. 

We have a set of (insert number) questions to guide our discussion. Your answers will be confiden- 
tial, and your names won’t appear anywhere in the evaluation findings. 

I’m going to be asking two kinds of questions. Some of the questions ask you to answer “yes” or 
“no,” or they ask you to rate how satisfied you are with a part of the program. I’ll also ask you to 
explain your answers. For the other kind of question, you’ll just be asked for your thoughts and 
opinions. If you don’t understand a question, please ask me to repeat it. 
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(Interview Instructions) 

I would like to ask you some questions about how well our literacy program is working. The ques- 
tions cover several different areas, some of which you may know quite a bit about, and others you 
may not know much about. 

This isn’t a test of how much you know about the program, or of how well you’re doing in the pro- 
gram. It’s a way to get information and ideas that we can use to see how the program’s doing, and 
to make changes if necessary. 

Your point of view is important. Please answer these questions as honestly as you can. All of your 
answers are confidential, and your name won’t be used anywhere in the evaluation, except to keep 
track of who we’ve talked to so far. 

There are two kinds of questions in this interview. Some of the questions ask you to answer “yes” 
or “no,” or they ask you to rate how satisfied you are with a part of the program. You can explain 
your answer to any of these questions. For the other kind of question, you’ll just be asked for your 
thoughts and opinions. 

There are (insert number) questions in total. If you don’t understand a question, please ask me to 
repeat it. 



C* 
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Sample items for use with tutors 

In what wavs does our program have a relevant and applied philosophy? 

1 . Do you think the way the program operates is consistent with its philosophy and vision? (2.1) 
Yes No Don’t know 

Please explain 



2. Were the program’s philosophy and vision communicated to you in ways that were easy to 
understand? (2.3) 

Yes No Don’t know 

Please explain 



. How satisfied are you with your opportunity to give input in developing the program’s mission, 
vision, and values statements? (2.6) 



Not at all 
satisfied 

1 



Very 

satisfied 



Doesn’t 

apply 



Please explain 
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4. How satisfied are you with your opportunity to give input when the program reviewed its mission, 
vision, and values statements? (2.7) 

Not at all Very Don’t 

satisfied satisfied know 

1 2 3 4 5 

Please explain 



In what ways do we deliver programming that addresses the needs of the individual? 

5. Did you get useful lesson-planning information from your student’s initial assessment? (3.3) 
Yes No Don’t know 

Please explain 



T-n move Hnpc \;r>nr etnHf»nt nnrtirinntp in Hefinina and revisiting hi s or her own goals? (3.61 

XXI V V nut V V uj U UV/VL1 J DbMtxv'wi. J-' — — Q O O ' ' 



Please describe 



7. In what ways does your student participate in his/her assessments? These include assessments 
during the program and, if possible, when the student leaves the program. (3.8) 

Please describe 
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8. When your student is assessed during the program, do you find the resulting information relevant in 
reviewing his or her progress? (3.8) 

Yes No Don’t know 

Please explain 



9. When your student is assessed during the program, do you use the resulting information in 
developing lesson plans? (3.8) 

Yes No Don’t know 

Please explain 



10. How often do you give your student praise and positive reinforcement? (3.18) 

Never Very Don’t 

often know 

12 3 4 5 

Please explain 



In what wavs are we providing quality tutoring? 

1 1 . In your opinion, is the tutor job description clearly written? (4.2) 

Yes No Don’t know 

Please explain 
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12. In your opinion, is the tutor job description realistic? (4.2) 
Yes No Don’t know 

Please explain 



13. The program should provide initial training for tutors in at least the following areas: introduction 
to literacy; student-centered approaches, methods and strategies in reading, writing and/or math; 
available resources and support; and tutor roles and responsibilities. How satisfied are you that the 
training you received addressed these topics? (4.4) 

Not at all Very Don’t 

satisfied satisfied know 

1 2 3 4 5 

Please explain 



14. How satisfied are you with other training provided by the program? (4.5) 



Not at all 








Very 


Don’t 


satisfied 








satisfied 


know 


1 


2 


3 


4 


5 





Please explain 



15. How available are the program staff when you have questions or concerns? (4.8) 

Not at all Very Don’t 

available available know 

1 2 3 4 5 

Please explain 
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16. How satisfied are you with the help the program staff gives you in using the program’s books and 
other resource materials? (4.10) 

Not at all Very Don’t 

satisfied satisfied know 

1 2 3 4 5 

Please explain 



17. How satisfied are you with the help the program staff gives you in improving your effectiveness as 
a tutor? (4.11) 

Not at all Very Don’t 

satisfied satisfied know 

1 2 3 4 5 



Please explain 



18. How satisfied are you with your opportunity to network with other tutors? (4.12) 

Not at all Very Don’t 

satisfied satisfied know 

1 2 3 4 5 

Please explain 



19. Do you feel valued and appreciated as a tutor? (4.13) 
Yes No Don’t know 

Please explain 
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20. How satisfied are you with the program’s recognition events for tutors? (4.14) 



Not at all 


Very 


Don’t 


satisfied 

12 3 4 


satisfied 

5 


know 



Please explain 



In what wavs do we have sound program administration? 

21 . Were the program’s expectations regarding confidentiality clearly communicated to you? (5.21) 
Yes No Don’t know 

Please explain 



22. How satisfied are you with the number of books and other resource materials for students and 
tutors? (5 .34) 

Not at all Very Don’t 

satisfied satisfied know 

1 2 3 4 5 

Please explain 



23. Do you find that books and other resource materials are available when you need them? (5.34) 
Yes No Don’t know 

Please explain 
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24. How satisfied are you that the books and other resource materials are relevant to you and your 
student? (5.35) 

Not at all Very Don’t 

satisfied satisfied know 

1 2 3 4 5 

Please explain 



25. The program’s facilities should be safe and inviting for all participants. How satisfied are you with 
the program’s facilities in this regard? (5.37) 

Not at all Very Don’t 

satisfied satisfied know 

1 2 3 4 5 

Please explain 



In what wavs is our program effectively planned and evaluated? 

26. How satisfied are you with your opportunity to take part when the program sets its goals and 
objectives? (6.9) 

Not at all Very Don’t 

satisfied satisfied know 

1 2 3 4 5 

Please explain 
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27. How satisfied are you with your opportunity to take part 
effectiveness? (6.10) 



in evaluating the program’s 



Not at all 
satisfied 



Very Don’t 

satisfied know 



1 2 3 4 5 



Please explain 



28. Have you been encouraged to give feedback on the program? (6.1 1) 
Yes No Don’t know 

Please explain 



Program Results 

29. How satisfied are you 



with the program overall? (#9) 



Not at all 
satisfied 



12 3 4 



Please explain 



Very 

satisfied 

5 



Don’t 

know 
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Sample Items for use with students 



Philosophy 

1 . The people in this program have developed guidelines and beliefs about how it should operate. 
(This is called the program philosophy.) Do you think the way it actually operates matches that 
philosophy? (2.1) 

Yes No Don’t know 

Please explain 



2. The people involved in this program have developed guidelines and beliefs about how it should 
operate. (This is called the program philosophy.) Did someone explain this philosophy to you in a 
way that was easy to understand? (2.3) 

Yes No Don’t know 

Please explain 



3. The people in this program have described how the program would look if it were the best it could 
be. They make their plans for the future of the program based on this picture. (This is called the 
program vision.) Did someone explain the vision to you in a way that was easy to understand? 
(2.3) 

Yes No Don’t know 

Please explain 



O 
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4. The people in this program decide on what values are important for this organization, what its 
purpose is (its mission) and what its future should be (its vision). How satisfied are you with the 
chance you had to take part in those decisions? (2.6) 

Not at all Very Doesn’t 

satisfied satisfied apply 



1 2 3 4 5 



Please explain 



5. The people in this program decide on what values are important for this organization, what its 
purpose is (its mission) and what its future should be (its vision). Every three years they review 
these things and see if they need to be changed. How satisfied are you with your chance to take 
part in that review? (2.7) 

Not at all Very Don’t 

satisfied satisfied know 

1 2 3 4 5 

Please explain ' 



Programming 

6. When you first joined the program, did you get useful information from your initial assessment? 
(3.3) (Note: you may wish to use “interview with the coordinator” rather than “initial 
assessment.”) 

Yes No Don’t know 

Please explain 
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7 . 



As a student, you are supposed to have the chance to take part in setting your own learning goals. 
In what ways do you take part in setting your own goals? (3.6) 



Please describe 



8. Your lesson plans and ways of measuring progress should be based on what you want to learn. 
How satisfied are you that this is what happens? (3.7) 

Not at all Very Don’t 

satisfied satisfied know 

1 2 3 4 5 

Please explain 




9. 



When your progress is measured, do you and your tutor use the information to figure out how 
much progress you have made toward your goals? (3.8) 

Yes No Don’t know 



Please explain 



10. How satisfied are you that the lessons you have with your tutor fit your goals, skills, and interests? 
(3.11) 

Not at all Very Don’t 

satisfied satisfied know 

1 2 3 4 5 

Please explain 
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1 1 .You should be able to talk to program staff to ask them questions or talk about any concerns. How 
satisfied are you that staff are available to you in this way? (3.14) 



Not at all 
satisfied 



Very 

satisfied 



Don’t 

know 



1 



Please explain 



12. How satisfied are you with the support services the program offers you as a student? (Note: We 
suggest you name some examples of what your program offers students so that this is not so 
abstract.) (3.16) 

Not at all Very Don’t 

satisfied satisfied know 

1 2 3 4 5 

Please explain 



13. The program should have information about different kinds of support services you could use 
outside the program. How satisfied are you with the help in finding support services outside the 
program? (3.16) 

Not at all Very Doesn’t 

satisfied satisfied apply 

1 2 3 4 5 

Please explain 
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14. When you want to find out about other education and training programs, someone in the literacy 
program should help you do that. How satisfied are you with this help? (3.17) 

Not at all Very Doesn’t 

satisfied satisfied apply 

1 2 3 4 5 

Please explain 



15. Do you feel encouraged and supported by your tutor? (3.18) 
Yes No Don’t know 

Please explain 



16. Do you feel encouraged and supported by program staff? (3.18) 
Yes No Don’t know 

Please explain 



17. Do you like the way the program recognizes the work you do and the progress you make? (3.19) 
Yes No Don’t know 

Please explain 
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Administration 

18. Is it clear to you what the program expects about keeping information private and confidential? 
(5.21) 

Yes No Don’t know 

Please explain 



19. How satisfied are you that there are enough books and other resource materials for you to use at 
the literacy program? (5.34) 

Not at all Very Don’t 

satisfied satisfied know 



1 2 3 4 5 



Please explain 



20. How satisfied are you that the program’s books and other resource materials are there when you 
want them? (5.34) 



Not at all 
satisfied 



Very 

satisfied 



Don’t 

know 



1 



Please explain . 



O 



Setting the Compass 



120 



page 117 



21 . The books and other resource materials should be interesting and related to what you want to 
learn. How satisfied are you that they are? (5.35) 

Not at all Very Don’t 

satisfied satisfied know 

1 2 3 4 5 



Please explain 



22. The program’s facilities should make people feel welcome and safe. How satisfied are you that the 
program’s facilities make these things possible? (5.37) 

Not at all Very Don’t 

satisfied satisfied know 

1 2 3 4 5 

Please explain 



Planning and evaluation 

23. The people involved in this program decide what the program’s goals are. How satisfied are you 
with the chance you had to take part in making those decisions? (6.9) 

Not at all Very Don’t 

satisfied satisfied know 

1 2 3 4 5 

Please explain 
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24. How satisfied are you with the opportunity you had to help evaluate the program? (6.10) 

Not at all Very Don’t 

satisfied satisfied know 

1 2 3 4 5 

Please explain 



25. Have you been encouraged to give feedback on the program? (6.1 1) 
Yes No Don’t know 

Please explain 



Program Results 

26. How satisfied are you that the literacy program taught you what you needed to learn? (#7) 

Not at all Very Don’t 

satisfied satisfied know 

1 2 3 4 5 

Please explain 



27. How satisfied are you that you can use what you learned in your daily life? (#7) 

Not at all Very Don’t 

satisfied satisfied know 

1 2 3 4 5 

Please explain 
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28. How satisfied are you that the quality of the services provided was good? (#7) 



Not at all 
satisfied 



Very 

satisfied 



1 



Don’t 

know 



Please explain 



29. How satisfied are you that the literacy program will help you to achieve your goals? (#7) 

Not at all Very Don’t 

satisfied satisfied know 

1 2 3 4 5 

Please explain 




30. Overall, how satisfied are you with the literacy program? (#7) 

Not at all Very 

satisfied satisfied 

1 2 3 4 5 



Please explain 



Don’t 

know 
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Sample items for use with board/council/committee members 



In what ways is our program community-based? 

1 . Were your responsibilities as a board/council/committee member clearly documented and 
communicated to you? ( 1 .4) 

Yes No Don’t know 

Please explain 



2. How satisfied are you that the board/council/committee’s annual goals and action plans support the 
goals of the literacy program? (1.6) 

Not at all Very Don’t 

satisfied satisfied know 

1 2 3 4 5 

Please explain 



3. How satisfied are you with the support the board/council/committee receives to help it accomplish 
its literacy-related goals? (1.7) 

Not at all Very Don’t 

satisfied satisfied know 

1 2 3 4 5 

Please explain 



Setting the Compass 



page 121 



Tn what wavs does our program have a relevant and applied philosophy? 

4. Do you think the program’s goals match its philosophy and vision? (2.1) 

Yes No Don’t know 

Please explain 



5. Do you think the program’s services and operations match its philosophy and vision? (2.1) 
Yes No Don’t know 

Please explain 



6. Were the program’s philosophy and vision communicated to you in ways that were easy to 
understand? (2.3) 

Yes No Don’t know 

Please explain 



7. How satisfied are you that the program’s philosophy-related statements reflect the literacy needs of 
the community? (2.4) 

Not at all Very Don’t 

satisfied satisfied know 

1 2 3 4 5 

Please explain 
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8. How satisfied were you with your opportunity to give input in developing the program’s mission, 
vision, and values statements? (2.6) 

Not at all Very Doesn’t 

satisfied satisfied apply 

1 2 3 4 5 

Please explain 



9. How satisfied were you with your opportunity to give input when the program reviewed its 
mission, vision, and values statements? (2.7) 

Not at all Very Don’t 

satisfied satisfied know 

1 2 3 4 5 

Please explain 



In what ways do we have sound program administration? 

10. The program’s fiscal management should include, at a minimum: an annual budget approved by 
the board/council/ committee; regular reports presented to the board/council/committee; and an 
annual review or audit. How satisfied are you that the program’s fiscal management includes these 
things? (5.5) 

Not at all Very Don’t 

satisfied satisfied know 

1 2 3 4 5 

Please explain 
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1 1. Were the program’s expectations regarding confidentiality clearly communicated to you? (5.21) 
Yes No Don’t know 

Please explain 



12. How satisfied are you that staff time is organized so that both the participants’ needs and the 
program’s administrative demands are met? (5.23) 

Not at all Very Don’t 

satisfied satisfied know 



1 2 3 4 5 



Please explain 



13. Do you feel valued and appreciated as a volunteer? (5.25) 
Yes No Don’t know 

Please explain 



14. How satisfied are you with the program’s recognition events for volunteers? (5.26) 

Not at all Very Don’t 

satisfied satisfied know 

1 2 3 4 5 

Please explain 
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to all participants? 

Don’t 
know 

1 2 3 4 5 

Please explain 



15. How satisfied are you that books and other resource materials are available 
(5.34) 

Not at all Very 

satisfied satisfied 



16. The program’s facilities should be safe and inviting for all participants and should have enough 
room for administrative tasks and for working with participants. How satisfied are you with the 
program’s facilities in this regard? (5.37) 

Not at all Very Don’t 

satisfied satisfied know 

1 2 3 4 5 

Please explain 



In what ways is our program effectively planned and evaluated? 

17. In your opinion, are the evaluation tools used by the program compatible with its philosophy, 
goals and delivery methods? (6.5) 

Yes No Don’t know 

Please explain 
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18 . 



How satisfied are you with your opportunity to give input when the program sets its goals and 
objectives? (6.9) 



Not at all Very Don’t 

satisfied satisfied know 



1 2 3 4 5 



Please explain 



19. How satisfied are you with your opportunity to take part in evaluating the program’s 
effectiveness? (6.10) 

Not at all Very Don’t 

satisfied satisfied know 

1 2 3 4 5 

Please explain 



20. Have you been encouraged to give feedback on the program? (6.11) 
Yes No Don’t know 

Please explain 



# 
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Sample Items for use with volunteer staff 

Tn what ways does our program have a relevant and applied philosophy? 

1 . Do you think the program’s services and operations match its philosophy and vision? (2.1) 

Yes No Don’t know 

Please explain 



2. Were the program’s philosophy and vision communicated to you in ways that were easy to 
understand? (2.3) 

Yes No Don’t know 

Please explain 



3. How satisfied were you with your opportunity to give input in developing the program’s mission, 
vision, and values statements? (2.6) 

Not at all Very Doesn t 

satisfied satisfied apply 

1 2 3 4 5 

Please explain 
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How satisfied were you with your opportunity to give input when the program reviewed its 


mission, vision, and values statements? (2.7) 






Not at all 


Very 


Don’t 


satisfied 


satisfied 


know 


1 2 3 


4 5 





Please explain 



In what ways do we deliver programming that addresses the needs of the individual? 

5. How knowledgeable are you about other education and training programs in the community?(3.17) 

Not at all Very Don’t 

knowledgeable knowledgeable know 

1 2 3 4 5 

Please explain 



6. Do you help students use the information available on other education and training programs? 
(3.17) 

Yes No Don’t know 

Please explain 



7. How often do you give students praise and positive reinforcement? (3.18) 
Never Very 

often 

1 2 3 4 5 

Please explain 



Don’t 

know 



0 
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In what ways do we have sound program administration? 

8. In your opinion, is your job description clearly written? (5.10) 

Yes No Don’t know 

Please explain 



9. In your opinion, is your job description realistic? (5.10) 
Yes No Don’t know 

Please explain 



10. Volunteer staff should have access to initial and ongoing training in areas relevant to their duties. 
How satisfied are you that the program has provided you with this training? (5.14) 

Not at all Very Don’t 

satisfied satisfied know 

1 2 3 4 5 

Please explain 



1 1 . How satisfied are you that the program’s expectations of you are clearly defined and 
communicated? (5.17) 

Not at all Very Don’t 

satisfied satisfied know 

1 2 3 4 5 

Please explain 
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12. How satisfied are you that you have the resources you need to do your work? (5.18) 

Not at all Very Don’t 

satisfied satisfied know 

1 2 3 4 5 

Please explain 



13. Were the program’s expectations regarding confidentiality clearly communicated to you? (5.21) 
Yes No Don’t know 

Please explain 



14. How satisfied are you that the staff’s time is organized so that both the participants’ needs and the 
program’s administrative demands are met? (5.23) 

Not at all Very Don’t 

satisfied satisfied know 

1 2 3 4 5 

Please explain 



15. Do you feel valued and appreciated as a volunteer? (5.25) 
Yes No Don’t know 

Please explain 
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16. How satisfied are you with the program’s recognition events for volunteers? (5.26) 



Not at all 
satisfied 



Very Don’t 

satisfied know 



1 2 3 4 5 



Please explain 



17. How satisfied are you that books and other resource materials are available to all participants? 
(5.34) 

Not at all Very Don’t 

satisfied satisfied know 

1 2 3 4 5 

Please explain 



18. The program’s facilities should be safe and inviting for all participants and should have enough 
room for administrative tasks and for working with participants. How satisfied are you with the 
program’s facilities in this regard? (5.37) 

Very Don’t 



Not at all 
satisfied 

1 



satisfied 

5 



know 



Please explain . 
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In what ways is our program effectively planned and evaluated? 

19. How satisfied are you with your opportunity to give input when the program sets its goals and 
objectives? (6.9) 

Not at all Very Don’t 

satisfied satisfied know 

1 2' 3 4 5 

Please explain 



20. How satisfied are you with your opportunity to take part in evaluating the program’s 
effectiveness? (6.10) 

Not at all Very Don’t 

satisfied satisfied know 

1 2 3 4 5 

Please explain 



21 . Have you been encouraged to give feedback on the program? (6.11) 
Yes No Don’t know 

Please explain 
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Sample items for use with paid staff 



In what ways is our program community based? 

1 . How satisfied are you that the board/council/committee’s annuals goals and action plans support 
the goals of the literacy program? (1.6) 

Not at all Very Don’t 

satisfied satisfied know 

1 2 3 4 5 

Please explain 



In what ways does our program have a relevant and applied philosophy? 

2. Do you think the program’s services and operations match its philosophy and vision? (2.1) 

Yes No Don’t know 

Please explain 



3. Were the program’s philosophy and vision communicated to you in ways that were easy to 
understand? (2.3) 

Yes No Don’t know 

Please explain 
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4. How satisfied were you with your opportunity to give input in developing the program’s mission, 
vision, and values statements? (2.6) 

Very Doesn’t 



Not at all 
satisfied 

1 



satisfied 

5 



apply 



Please explain 



5. How satisfied were you with your opportunity to give input when the program reviewed its 
mission, vision, and values statements? (2.7) 

Not at all Very Don’t 

satisfied satisfied know 

1 2 3 4 5 

Please explain 



In what ways do we deliver programming that addresses the needs of the individual? 

6. Are you satisfied that the information gathered during the students’ initial interviews is handled 
confidentially? (3.1) 

Yes No Don’t know 

Please explain 



7. In what way do students participate in their initial assessments? (3.3) 



Please describe 
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8. In what ways do students participate in defining and revisiting their own goals? (3.6) 
Please describe 



9. In what ways do students participate in their ongoing and exit assessments? (3.8) 
Please describe 



10. How knowledgeable are you about other education and training programs in the community? 
(3.17) 

Not at all Very Don’t 

knowledgeable knowledgeable know 

1 2 3 4 5 

Please explain 



1 1 . Do you help students use the information available on other education and training programs? 
(3.17) 

Yes No Don’t know 

Please explain 
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12. How often do you give students praise and positive reinforcement? (3.18) 
Never Very 

often 

1 2 3 4 5 



Please explain . 



Don’t 

know 



In what ways are we providing quality tutoring? 

13. How satisfied are you with the opportunities you have to help tutors improve their effectiveness? 
(4.11) 

Not at all Very Don’t 

satisfied satisfied know 

1 2 3 4 5 

Please explain 



14. How satisfied are you with your ability to help tutors improve their effectiveness? (4.1 1) 



Not at all 
satisfied 

1 



Very 

satisfied 



Don’t 

know 



Please explain 
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In what ways do we have sound program administration? 

15. How satisfied are you that your job description is comprehensive? (5.8) 



Not at all 
satisfied 



Very 

satisfied 



1 



Please explain 



Don’t 

know 



16. How satisfied are you that your job description is clearly written? (5.8) 

Not at all Very Don’t 

satisfied satisfied know 

1 2 3 4 5 

Please explain 



17. How satisfied are you that your job description is realistic? (5.8) 



Not at all 
satisfied 

1 



Very 

satisfied 



Please explain . 



Don’t 

know 
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18. Depending on their roles in the program, staff should have access to training in the following 
areas: teaching reading, writing and math; volunteer management; assessment techniques and 
tools; program and office management; building community partnerships; and current literacy 
research and trends. How satisfied are you with your access to training in these areas? (5.13) 

Not at all Very Don’t 

satisfied satisfied know 

1 2 3 4 5 

Please explain 



19. The program should provide staff with networking opportunities at least twice a year through its 
links with other literacy organizations. How satisfied are you with the opportunity to network with 
others working in literacy? (5.16) 

Not at all Very Don’t 

satisfied satisfied know 

1 2 3 4 5 

Please explain 



20. Were the program’s expectations regarding confidentiality clearly communicated to you? (5.21) 
Yes No Don’t know 

Please explain 
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21. How well do you think the program’s recruitment plans take into consideration the space and 
resources available? (5.22) 

Not at all Very Don’t 

well well know 

1 2 3 4 5 

Please explain 1 



22. How satisfied are you that the staff’s time is organized so that both the participants’ needs and the 
program’s administrative demands are met? (5.23) 

Not at all Very Don’t 

satisfied satisfied know 

1 2 3 4 5 

Please explain 



23. How satisfied are you that participant records are relevant and up-to-date? (5.28) 

Not at all Very Don’t 

satisfied satisfied know 

1 2 3 4 5 

Please explain ! 
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24. How satisfied are you that books and other resources are available when participants need them? 
(5.34) 

Not at all Very Don’t 

satisfied satisfied know 

1 2 3 4 5 

Please explain 



25. The program’s facilities should be safe and inviting for all participants and should have enough 
room for administrative tasks and for working with participants. How satisfied are you with the 
program’s facilities in this regard? (5.37) 

Not at all Very Don’t 

satisfied satisfied know 

1 2 3 4 5 

Please explain 



26. How satisfied are you that the program provides you with the office and instructional equipment 
you need to carry out your responsibilities? (5.38) 

Not at all Very Don’t 

satisfied satisfied know 

1 2 3 4 5 

Please explain 
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In what wavs is our program effectively planned and evaluated? 

27. Do you keep informed of relevant development in literacy practice (e.g. STAPLE, Lit-Link, etc.)? 
Yes No Don’t know 

Please explain 



28. Do you incorporate relevant developments in literacy practice into program planning? (6.3) 
Yes No Don’t know 

Please explain 



29. In your opinion, are the evaluation tools used by the program compatible with its philosophy, 
goals and delivery methods? (6.5) 

Yes No Don’t know 

Please explain 



30. Hqw satisfied are you with your opportunity to give input when the program sets its goals and 
objectives? (6.9) 

Not at all Very Don’t 

satisfied satisfied know 

1 2 3 4 5 

Please explain 
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31 . How satisfied are you with your opportunity to take part in evaluating the program’s 
effectiveness? (6.10) 

Not at all Very Don’t 

satisfied satisfied know 

1 2 3 4 5 

Please explain 



32. Have you been encouraged to give feedback on the program? (6.11) 
Yes No Don’t know 

Please explain 
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Sample items for community agencies 

In what ways is our program community based? 

1 . How satisfied are you that the services offered by the literacy program are appropriate for the needs 
of the community? (1.2) 

Not at all Very Don’t 

satisfied satisfied know 

1 2 3 4 5 



Please explain 



Other items suggested as useful by pilot site programs: 
1 . Do you know about the literacy program? 

Yes No Don’t know 




Do you have information about the program? 
Yes No Don’t know 



3. Do you refer people to the program? 

Yes No Don’t know 

If yes, what do you refer them for? (e.g. tutoring, information about literacy, to tutor, etc.) 



If no, why not? 



4. Have you been contacted by program staff in the last year? 
Yes No Don’t know 

If yes, what for? How did you respond? 
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Appendix C 



Case study 



Purpose of the case study 

This case study is based on the combined experiences of the seven literacy programs that piloted Set- 
ting the Compass. It is intended as an example of a volunteer literacy program using the process out- 
lined in the document. You may find it useful to refer to as you work through the document. 

Description of the program 

The ABC Literacy Program has been in operation nine years and Jane Doe has been the coordinator 
for almost four years. The program is located in Anywhere, a town of eight thousand surrounded by 
farm and ranch land. Although most people are involved in small businesses or prairie agriculture, 
other major employers are an oilfield supply company, a hospital and an extended care facility. 

Most of the program's participants live in town, but some students and tutors come from the surround- 
ing area, including from two nearby smaller villages, Smalltown and Smallertown. All in all, the pro- 
gram serves a population of just over 16,500 people, according to the latest census. The program is 
open from September through the middle of June and takes new students and volunteers on an on- 
going basis. 

ABC has twenty-three pairs, eight of them new matches this year. Most pairs work on basic literacy 
and numeracy at a variety of different levels, and six focus on ESL. Some tutors work one-on-one 
with more than one student. Since the oilfield supply company opened, the program has been getting 
quite a few students who work there, but Jane finds that they are often sent out of town on short 
notice. The program began ESL classes last year, but found the scheduling difficult because so many 
of the students own businesses and cannot get to class at a set time. ABC also offers classes through 
the Adult Learning Council which focus on a variety of topics, such as spelling or doing a job search. 

Last year, the program planned a Books for Babies project which will get off the ground this year with 
money from Anywhere Family and Community Support Services and the Somewhere Regional 
Health Authority's new funding initiative. Also this year, the program intends to begin offering regular 
in-service workshops to its volunteer tutors, with plans for five of them already made. 

Jane Doe, as coordinator, is the only paid staff member of the program. She works part-time, sixteen 
hours a week, and frequently puts in volunteer hours herself. Her job is made easier because she has a 
.very supportive group of tutors. Many find the time to do extra volunteer work for the program, flip- 
ping burgers at a fundraiser or shelving library books, for example. Though Jane has tried to encour- 
age the students to get involved in the program beyond their tutoring sessions, she has not had much 
success in finding a fit for them. 

The program has an active advisory committee. They meet four to six times a year, depending on the 
program's activities, and are especially useful in helping to promote the program. The Community 
Adult Learning Council is the program's hosting authority. 
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Step 1 

Following the guidelines as laid out in Setting the Compass, Jane began by reviewing the whole docu- 
ment. The most difficult part of this task and, as it turned out, for other parts of the process as well, 
was finding uninterrupted time. She ended up taking the document home to read it one afternoon. 
When she got back to the office, she had a volunteer go through the document and photocopy the 
worksheets, to make sure there were originals left to use in future years. 

Reviewing the document gave her a sense of what was to come and was useful when she approached 
people to be on an evaluation team because she could answer their questions. She and John Smith, the 
chairperson of the program's advisory committee, brainstormed possible evaluation team members. 
They began by listing the stakeholders whom they wanted to see represented. They listed more names 
than they would need in case the first few contacted were unable to participate. 

Jane then used the sample letter as a guide and wrote to the first four potential members on the list. 
She knew the people fairly well and could have just called them, but she felt it was easier to write a 
letter than to explain to each one individually why she was contacting them. She followed up on the 
letter with a phone call several days later. Two of the four had too many commitments already and so 
she went back to the list for another two names. Jane and John really wanted to see a student on the 
team and, in the end, a student-tutor pair agreed to join. 

Before the first evaluation team meeting in late September, Jane completed the portion of the docu- 
ment which describes the program and its context. Most of the information was readily available and 
the process took about an hour and a half. She started a separate file of copies of the pages which 
she'd send to Community Programs when they called for them. 

The members of the evaluation team met for the first time the next week. After the basic introduc- 
tions, Jane gave them a quick overview of the document and the various purposes of each step. This 
process allowed the team to be sure everyone understood the tasks ahead. 

The members of the evaluation team agreed that, before their next meeting, they would like to see the 
results of the checklist which determined how ready the program was for an evaluation. They felt that 
it was up to the program’s advisory committee to deal with any barriers to evaluation so that they 
could proceed with their tasks. Jane sent the form to the tutors, with a cover letter explaining that the 
program wanted to undertake an evaluation in the near future and would use their responses to deter- 
mine how ready the program was for it. She also sent the form to advisory committee members and 
gave it to her two office volunteers when they came in. Once compiled, the results showed that several 
points on the checklist needed to be addressed. However, the advisory committee decided that each of 
these could be dealt with through communication and future planning and decided to go ahead with 
the evaluation. 

Step 2 

When the evaluation team members arrived for their next meeting, Jane handed out copies of the 
checklist “Clarify why you're doing evaluation, and for whom.” There was quite a bit of discussion 
about who would be interested in the results of their evaluation and why. The difficulty was not in 
“choosing who” so much as in prioritizing them for this year's evaluation, because it was soon obvi- 
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ous that they couldn't do everything for everybody. They decided that for their first systematic evalua- 
tion, they wanted information that would be useful to students and tutors, and for basic accountability 
to the funder. By the time the group was finished with this exercise, the evaluation already seemed 
more focused. 



The group then went on to look at the suggested evaluation questions. The synopsis accompanying 
each question helped them to understand what parts of the program the questions were related to. 
They found that in some cases one question answered the needs of more than one audience. For 
example, determining the impact of the program on its participants would be a question funders 
would want answered, but it was also something the program coordinator and advisory committee 
always wanted to know and would be of great interest to tutors and students. 



It was obvious that for the audiences they had identified, they would want to focus on questions 3, 4, 
5, and 7 for this evaluation. Following the instructions for this section, they then looked in Appendix 
A at the standards and Program Results items associated with each of these questions. It would have 
been easy to choose the standards that they knew were already being achieved, but instead they chose 
standards that would be the most useful and appropriate for their program. Some were in fact ones the 
program already met. For others, they needed to collect information to find out how their program 
was doing. When they finished this exercise, the group had identified eighteen standards and question 
#7 as items that they were interested in working with. (Jane had already explained to the group that, 
together with the two pages in Step 1 , question #7 was to be submitted to Community Programs in 
place of the Participation Summary previously sent out by the funder.) 



The group found that working through Appendix A and making decisions about how they would col- 
lect information related to these items was time consuming but worthwhile. They found they had to 



think ahead about what would work best for their program, especially for their students and tutors. 
They decided they would hold a focus group with students and use a mail-out survey with tutors. Jane 
would also have to review program records for some of the standards. 



Once the evaluation team began identifying the resources needed for the standards they had selected, 
they realized it wasn't possible to deal with as many standards as they had chosen. They had to go 
back to the list and set priorities. They decided to work with a total of ten standards for this first eval- 
uation. That number seemed reasonable in that the amount of work involved was feasible and would 
provide enough information to make the evaluation worthwhile. 



Earlier in the process, when looking at how ready ABC was for evaluation in Step 1, everyone had 
agreed that they needed to decide how much to include in the budget for evaluation. Now that they 
were looking at the resources they needed, the evaluation team came up with a list of questions for 
Jane to consider. When did she have time to do evaluation-related work? If she really didn’t have any 
time, how could she make time? How much would it cost to make time, e.g. could she give up some 
other program activities? Could her paid hours be increased? Could the program pay someone else to 
do some of her current work? Thinking about these questions helped the program to estimate the cost 
of the evaluation in financial terms. 
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In the end, the evaluation team decided that the following would be the focus of their evaluation: 

• they would find out about overall student satisfaction by holding a focus group (part of #7 
in Appendix A); 

• they would find out about programming that addresses the needs of the individual by 
reviewing documents, holding the student focus group, and surveying tutors (standards 3.3, 
3.6, 3.8 and 3.12 in Appendix A); 

• they would find out in what ways the program provides quality tutoring by reviewing docu- 
ments and surveying tutors (standards 4.2, 4.4, 4.6 and 4.11); and 

• they would find out in what ways they have sound program administration by reviewing 
documents and polling advisory committee members at their next meeting (standards 5.1 
and 5.5). 

• Jane would also need to review student records and some other files to complete question 
#7 for Community Programs. 

They felt this information would be a good start to reviewing the program as a whole, dealing as it did 
with aspects of the program which seemed crucial to them. The team agreed the eight standards that 
they had taken off the list should be kept on file for the next evaluation cycle. 

Because the members of the evaluation team had been involved from the start in making decisions 
about the planning and evaluation process, they were very willing to take on responsibilities when it 
came to filling out the workplan in Step 2. They would need to find some additional volunteers to 
work with them, and two of the team members agreed to make sure that it was not left on Jane's shoul- 
ders. The group ended Step 2 with a clear sense of purpose and a realistic picture of what they needed 
to do. 

By mid-November, the evaluation team informed the rest of the program participants about the evalu- 
ation they had planned. Jane wrote an article for the program newsletter and asked the tutors to inform 
their students about the evaluation, its purpose, and how people would be involved. Because system- 
atic evaluation had never been part of ABC before, some people assumed that “something was 
wrong” for it to be happening now. This highlighted for Jane and the others involved that planning 
and evaluation needed to be presented as a regular part of program practice in order for people to feel 
comfortable giving feedback on the program. 

Step 3 

For Step 3, Jane reviewed the methods of evaluation which the program had used in the past, which 
consisted primarily of the evaluations handed out at the end of tutor training sessions and compiling 
statistics for the government. She also worked with another evaluation team member to review docu- 
ments. This activity highlighted the program's need to have information recorded rather than just rely- 
ing on people’s memories. They realized that while a lot of the necessary documentation can be quite 
informal (just a quick note to describe a conversation with a potential community partner, for exam- 
ple), it does need to be systematic. 

The evaluation team found it very straight-forward to develop the tutor survey and the student focus 
group questions using the items in Appendix B, and tested the tutor surveys by using process recom- 
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mended in Step 3. The town librarian, a friend to the literacy program but not directly connected to it, 
had agreed to facilitate the student focus group. Jane called several students and, having explained the 
purpose of the evaluation, checked to see if they would be willing to test the questions that were going 
to be asked in the focus group. The librarian then called the three students who had agreed. She found 
that the students asked questions about both the questions and the process, which helped her prepare 
for the focus group. 

Because they were only looking at one standard that required getting input from the advisory commit- 
tee, there was no reason to develop a survey or interview form. Jane asked to have the question relat- 
ing to standard 5.5 (fiscal management) put on the agenda for the next advisory committee meeting. 

The team decided that for this first evaluation, it would be worth the time to have Jane contact stu- 
dents to invite them to the focus group and to explain why questions were being asked of them. Jane 
reminded the students that their perspective is unique and valuable. During that call and during the 
actual focus group, Jane and the librarian explained that the evaluation was not to judge the tutors' or 
coordinator's performance, but rather to see how the program was doing as a whole, e.g. whether it 
was providing adequate tutor training and whether the student assessments were useful. 

The evaluation team used the focus group to get information on the students' overall satisfaction with 
the program, part of question #7 in Program Results. The rest of #7, though, required statistical data 
which was easiest for Jane to access. She filed the pull-out sheets from question #7 with the earlier 
“Participation Summary” pages from Step 1 , ready to fill out when she received notice from Commu- 
nity Programs at Advanced Education and Career Development. 

As feedback from participants began coming in through March and April, Jane kept track of it by sim- 
ply having three different files, one for students, one for tutors, and one for advisory committee mem- 
bers. She developed a log in which she recorded how many surveys had been mailed out and how 
many returned. 

Step 3 was the most time-consuming so far, but well worth the effort. The evaluation team was 
pleased that, having planned early, they had the time to prepare and test the tools without feeling too 
rushed and that they had six weeks or more to bring in all the results. The turn-out for the focus group 
had been quite small, but they recognized that it was the first time and that it would get easier each 
time as it became part of the program. 

Step 4 

By now the evaluation team had been working together for several months, and had met four times. 
They found they needed to readjust the timelines a few times and had to resolve some disagreements 
about how best to proceed, but on the whole they had accomplished much and learned a lot. They had 
made a habit of meeting in Jane's office over lunch, and had enjoyed many laughs. And, in just over 
four weeks, by the end of May, they planned to have completed the first evaluation of ABC. 

Jane compiled the information from the focus group and surveys. It took time, but she found it inter- 
esting and informative. She liked the fact that the questions presented in Appendix B presented a bal- 
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ance of easily-tallied numerical information and longer descriptive comments. She made note of 
several good quotes to use when telling others about the program and the evaluation. 

Jane found that calculating the average of each rating scale went very quickly, but summarizing the 
descriptive comments took longer because it was an unfamiliar task. She went through all the feed- 
back once and highlighted key ideas, and found this helped to come up with a code. 

Once Jane finished compiling the collected information, the evaluation team met again. Together, they 
worked to outline the factors which contributed to or impeded the program's success. As with any 
town, lots happened that year in Anywhere. At first the team was tempted to see everything as a con- 
tributing or impeding factor. After discussion, though, the team members settled on two factors which 
had significantly affected their program that year: the new funding available through the regional 
health authority and the short time many students spent in the program because of local employment. 

The evaluation team found the next part of the process easy to do. They used the chart provided to 
note all the information that had been summarized for each standard from document reviews, the 
focus group, interviews and surveys. From there, they decided which standards they had achieved and 
which needed work. On the whole, the experience was positive and reassuring. It showed them clearly 
what needed to be done and, just as importantly, showed many of the ways the program was already 
doing well. Much of the information validated what the team thought about the program already. 
They were pleased to have their hunches confirmed. 

As they began to answer the evaluation questions, the team talked about how long ago November now 
seemed. It was useful and interesting to apply the standards and the collected information to the ques- 
tions selected six months before, bringing the process full circle. Answering the questions this specif- 
ically gave the team, especially Jane, very clear thoughts and ideas which could be used in 
presentations to stakeholder, including the next tutor training and her speech planned for the Rotary 
Club. 

They were glad, at this point, to have had the discussions about the additional factors which contrib- 
uted to and impeded their success. They realized, because of that conversation, why on-going assess- 
ment was a weaker area in programming than, say, initial assessment: many students did not stay in 
the program very long and left without notice. The program did need to look at on-going assessment, 
but looking at the contributing factor helped them put this particular program evaluation result in per- 
spective. 

On the whole, the team concluded that the program was meeting its goals. The students were satisfied 
overall. The areas where practice could be improved were clearly outlined. The very fact that the pro- 
gram was able and willing to respond to these was seen as a strength. They decided to write an article 
for the ABC Newsletter outlining the evaluation results and the conclusions they had drawn. Their 
conclusions would also be the basis of the upcoming planning day. 

As a result of the evaluation, the whole team and Jane especially felt focused, with clear and achiev- 
able goals in mind. The information they collected would be useful not only in moving their program 
forward and in reporting to Community Programs, but in reminding them to celebrate their successes. 
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Appendix 0 


Background on ALPS 


Overview 


The Alberta Literacy Project Standards (ALPS) Project was: 

• a two-phased project funded by the National Literacy Secretariat 
(NLS) of Human Resources Development Canada; 

• sponsored by the Association of Literacy Coordinators of Alberta 
(LCA) and managed by the Association’s Board of Directors; 

• jointly coordinated by Sharon Skage of Edmonton and Mamie Scha- 
etti of Calgary. 



Project Stakeholders While the standard membership of the LCA has had final say in the 

approving the results of the project, the following are all considered to be 
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stakeholders in ALPS, as they are in Alberta’s literacy programs more 

generally: 

• literacy coordinators 

• the LCA Board of Directors 

■ tutors 

■ students 

■ funders 

■ communities at large 

In the first phase of ALPS. 

■ in cooperation with stakeholders and the ALPS Advisory Committee, 
to develop good practice statements to serve as guidelines for effective 
community-based volunteer literacy programs in Alberta; 

• to ensure that the statements mesh with the Department of Advanced 
Education and Career Development’s accountability initiative, holding 
the needs of literacy programs paramount; 

• to have the statements approved by the standard membership of the 
LCA; 

• again in cooperation with stakeholders and the ALPS Advisory Com- 
mittee, to develop a set of standards which all programs should meet 
and be given the resources to meet; 

• to have the standards approved by the standard membership of the 
LCA. 
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In the second phase of ALPS: 

• in cooperation with stakeholders and the ALPS Advisory Committee, 
to develop appropriate methods of evaluating community-based vol- 
unteer literacy programs based on the good practice statements and 
standards approved in Phase One; 

• to field-test the evaluation tool in various literacy programs; 

• to have the evaluation tool ratified by the standard membership of the 
LCA; 

• to use the information gathered through the ALPS Project to recom- 
mend criteria for establishing base fundino. 

Results In the first phase: 

the Good Practice Statements for Volunteer Literacy Programs in 
Alberta were completed in the spring of 1997; 

• in June 1997, the standard membership of the LCA voted 98% in favor 
of approving the Good Practice Statements as guidelines for effective 
practice in volunteer literacy programs; 

• the Standards for Volunteer Literacy Programs in Alberta were com- 
pleted in March 1998; 

• in March 1998, the standard membership of the LCA voted 98% in 
favor of approving the Standards as benchmarks for use in evaluating 
and developing volunteer literacy programs. 

In the second phase: 

• Setting the Compass was completed in May 1999; 

• in May 1999, the standard membership of the LCA voted 88% in favor 
of approving Setting the Compass as a tool for program development 
and evaluation in volunteer literacy programs in Alberta. 



For more information 

Association of Literacy Coordinators of Alberta 
332 - 6th Avenue S.E. 

Calgary, Alberta 
T2G 4S6 

(403) 297-4995 telephone 
(403) 297-4849 fax 
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Appendix E 



Good Practice Statements and Standards 
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Introduction 



This document contains two of the three components of the Alberta Liter- 
acy Program Standards (ALPS) project. The good practice statements — 
the first component — are intended to be used as general guidelines or 
recommendations for practice in Alberta’s volunteer adult literacy pro- 
grams. The standards — the second component — identify the specific 
details of operation that program administrators can use as benchmarks 
when assessing the effectiveness of their programs. The third component, 
which has yet to be developed, will be an evaluation strategy that pro- 
grams can use to show how they are working towards achieving the stan- 
dards. 

It should be pointed out that the tools in this document are meant to 
address the common elements of Alberta’s literacy programs, namely 
those related to one-to-one tutoring. The Association of Literacy Coordi- 
nators of Alberta (LCA) recognizes that many programs deliver other ser- 
vices to meet their communities’ literacy needs, but commissioned this 
work to develop tools for volunteer adult literacy programs. 



The good practice statements and standards are “Alberta-grown”; they 
are an expression of what literacy coordinators across the province have 
described as characteristics of a high quality program. They will assist in 
program development, affirming what we already know about the exper- 
tise and successes in our programs, and helping us to identify areas where 
we can improve our practice. They will also be useful in demonstrating 
the value of our programs to our participants, communities, and funders. 
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Statement of 
beliefs 



The following is a summary of what literacy coordinators in Alberta 
described in 1996 as the beliefs and values that they bring to their work. 
These beliefs and values are essential to understanding and interpreting 
the good practice statements and standards. 

1. Literacy programs should reflect and meet the needs of the individual 
in his/her context (home, work, community). 



2. Accessible literacy education is a basic right. 

3. Literacy skills are empowering, and enhance a person’s quality of life. 

4. Every individual has worth and ability, and deserves to be respected. 

5. Learning happens when a person feels safe, is willing to take risks, 
and can feel personal success. 

6. Learning is lifelong. We are all both teachers and learners. 

7. Literacy is a community responsibility. 

8. Literacy coordinators are facilitators, helping others so they can help 
themselves. 

9. Honesty, sincerity, and fairness are essential values for a literacy pro- 
gram. 
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Criteria for 

developing 

standards 



Part of the process of developing these standards has been to test them for 
appropriateness. The following criteria were developed for this purpose 
and applied to each draft standard. 

Useful: Standards serve a useful purpose. 

Realistic: Standards relate to areas over which the program has control. 

Meaningful: Standards are meaningful and significant in each program's 
particular context. 

Mission related: Standards are related to the program missions identified 
in the ALPS consultations with coordinators in 1996. 

Performance related: Standards describe the desired process or out- 
come against which performance can be measured. 

Independent: Standards are independent; that is, one standard does not 
cover exactly the same component as another. 

Manageable: Standards are supported by information that is accessible 
and easily analyzed. 

Constructive: Standards serve the needs of the local program first. 
Clearly stated: Any potentially ambiguous terms are clearly defined. 
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Program 

missions 



In the 1996 ALPS consultations, volunteer literacy program coordinators 

were asked to describe their programs’ missions, or reasons for being. 

Their responses are summarized as follows. 

Volunteer literacy programs exist to: 

• support learners in achieving their literacy goals through the use of 
volunteer tutors; 

• provide flexible opportunities to improve reading, writing, math skills, 
and other areas of learning; 

• provide opportunities that support people in making positive changes 
for themselves, their families, and their communities; 

• respond to the literacy needs of both individual and agency members 
of their communities. 



Organization of 
the document 



The good practice statements and standards have been organized into a 
framework that corresponds to areas of program operation. To help you 
locate information quickly, the statements and standards are arranged in 
the following six categories: 

1. Community-based Programs 

4 . Relevant and Applied Philosophy 

5. Individualized Programming 

6. Quality Tutoring 

7 . Sound Program Administration 

8. Continuous Program Development 

In the pages that follow, each category heading is followed by the recom- 
mended good practice statements and the standards. 
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For the purpose of this document, the following definitions are being 
used: 

Benchmark: a point of reference in measuring or judging quality. 

Category: a general area of program operation used to organize good 
practice statements and standards. 

Community: the geographic area in which a program is located, whether 
it is as small as a neighborhood or as large as a municipal district, includ- 
ing the general population of that area; or more specific groups of indi- 
viduals and agencies within the larger population whose interests and 
activities are related to that of the literacy program. 

Evaluation: the process of collecting and analyzing data about a pro- 
gram or any of its components which program participants use to judge 
the effectiveness of that program relative to its stated goals. 

Exit assessment: the means of determining the extent to which a student 
has achieved his or her goals. (Also called final assessment.) 

Good practice: the conditions that must be created and the actions that 
must take place for the program to operate effectively and deliver high 
quality service. 

Good practice statement: a description of an activity or condition that 
contributes to an effective, high quality literacy program. 

Indicators: key pieces of data used to indicate whether a program is 
meeting the standards of performance. 

Initial assessment: the means of determining an individual student's 
strengths and levels of competency in reading, writing and/or math, and 
his or her preferred learning style. Initial assessment information is used 
to develop an instructional program which meets a student's needs, inter- 
ests, and goals. (Also called placement assessment.) 

Mission statement: a brief statement which describes the purpose of the 
literacy program. 

On-going assessment: the means of determining a student's progress 
toward his or her goals. (Also called progress assessment.) 
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Program: an individual volunteer literacy organization. When a standard 
says that “the program” is responsible for an activity or condition, the 
focus is on ensuring that the activity or condition takes place. Who actu- 
ally carries out the task depends on the individual program; it may be the 
literacy coordinator, another paid staff, the board or advisory council, or a 
volunteer. The assumption is that this decision will be made at the pro- 
gram level, and documented in job descriptions. In this way, responsibil- 
ity is not automatically placed on the coordinator if there are other 
program participants who take part in the program's operation. 

Program participants: those individuals directly involved in the pro- 
gram, including students, tutors, paid and volunteer staff and advisory 
council or board members. 

Qualitative data: facts, claims, and assertions in narrative form, and not 
in numbers. 

Quantitative data: facts, claims and assertions presented in numerical 
forms. 

Sources of data: the records, reports, or other materials that provide 
indicator information. 

Staff: includes both paid personnel and the volunteers who support the 
program in ways other than tutoring. 

Standard: a benchmark against which programs can be evaluated. 

Values statement: a summary of the ethical principles which underlie 
program practice. 

Vision statement: the picture that program participants develop of their 
organization in the future, describing the ideal to which their program 
aspires. 

Volunteer literacy programs: the provision of student-centered literacy 
education services to communities through the use of volunteer tutors. 
Although services may frequently include broader social and communi- 
cation skills, the focus is on improving reading, writing and math skills. 

Volunteers: those who donate time and energy to help achieve the objec- 
tives of the literacy program, including tutors, volunteer staff, and advi- 
sory council or board members. 
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Good practice statements and standards 
for volunteer literacy programs 

PREAMBLE 

Each of the following good practice statements and standards is based on 

the understanding that: 

• Students are the first priority in all aspects of literacy programming 
and operation. 

• Volunteer literacy programs respect the diversity of experiences, back- 
grounds, and goals of participants. 

• Programs must receive stable and adequate funding in order to accom- 
plish and/or maintain good practice. 



I. COMMUNITY-BASED PROGRAMS 



Good practice statement 


Standard 


a)The literacy program 
provides services which 
respond to its own com- 
munity’s literacy needs. 


i. The program assesses the literacy needs of its community at least 
once every three years. 

ii. Taking into account the program’s philosophy, vision, and avail- 
able resources, the program responds to identified needs with 
appropriate services. 


b)The program has a board, 
advisory council, or sim- 
ilar committee whose 
members represent the 
community. Members 
have a responsibility to 
be informed, active, and 
supportive; 


i. The recruitment process for board, council, or committee mem- 
bers is designed to gain representation from a broad range of com- 
munity groups and program participants. 

ii. The expectations and responsibilities of the board, council, or 
committee are clearly documented and communicated to members 
and prospective members. 

iii. The board, council, or committee is provided with the Literacy 
Coordinators of Alberta standards for literacy programs and with 
the expectations of the funder(s). 

iv. Working with program staff, the board, council, or committee 
establishes annual goals and action plans that support the goals of 
the literacy program. 

v. The program ensures its board, council, or committee receives the 
support it needs to accomplish its literacy-related goals. 

vi. The board, council, or committee meets at least two times per year 
to focus specifically on matters relating to the literacy program. 
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Good practice statement 


Standard 


c)The program develops 
and maintains partner- 
ships with community 
groups, agencies, and 
individuals. 


i. The program seeks out partnerships with community organiza- 
tions and individuals with whom it shares complementary goals. 

ii. The program acts as a community resource for literacy develop- 
ment. 




iii.The program reviews its community partnerships annually. 


d)The literacy program 
actively promotes the 
value of its services and 
seeks the support of its 
community. Community 
support is appropriately 
recognized. 


i. Promotional activities include efforts to reach potential students 
and tutors and a broad range of community groups. 

ii. The program recognizes community support, and reviews its 
methods for recognizing its supporters each year. 



2. RELEVANT AND APPLIED PHILOSOPHY 



Good practice statement 


Standard 


a)The literacy program has 
a clear philosophy and 
vision that guide pro- 
gram practice. 


i. The program ensures that all aspects of program practice are con- 
sistent with its philosophy and vision. 

ii. Philosophy- and vision-related documents are written in plain lan- 
guage. 

iii. The program’s philosophy and vision are communicated to all 
new staff, students and volunteers in ways that are easy to under- 
stand. 


b)The philosophy includes 
relevant mission and val- 
ues statements. 


i. Philosophy-related statements reflect the literacy needs of the 
community. 

ii. The program has documented mission and values statements. 


c)Program participants 
develop and periodi- 
cally update the vision, 


i. The program offers staff, students, and volunteers the opportunity 
for input when developing its vision, mission, and values state- 
ments. 


mission, and values 
statements. 


ii. At least every three years, the program offers staff, students and 
volunteers the opportunity for input when reviewing the vision, 
mission, and values statements. 
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3. INDIVIDUALIZED PROGRAMMING 



Good practice statement 


Standard 


a) The program provides 
the student with an inter- 
view and appropriate 
assessment prior to 
being matched with a 
tutor. 


i. The program provides each student with a confidential interview 
as soon as possible after he or she contacts the program. 

ii. The interview is used primarily to explain the program’s services 
and to discuss any expectations the student may have. 

iii. The program provides each student with an initial assessment 
which 

• is student-centered and participatory; 

• helps the student to set realistic literacy-related goals; 

• provides useful information for the student and for initial 
lesson planning; 

• establishes a baseline for determining progress. 


b)Staff give new students 
an orientation to the pro- 
gram. 


i. A staff member clearly explains the student’s basic responsibilities 
and asks him or her to make a commitment to those responsibili- 
ties. 

ii. A staff member or volunteer introduces the student to the pro- 
gram’s facilities and operations. 


c) The literacy program 
assists the student in 
progressing toward his 
or her goals. The stu- 
dent, tutor, and coordi- 
nator work together to 
set realistic goals, moni- 
tor learning and deter- 
mine progress. 


i. Each student participates in defining and revisiting his or her own 
goals. 

ii. The student’s goals form the focus of lesson planning and the 
basis for on-going and exit assessment. 

iii. The program provides each student with on-going and, if possible, 
exit assessment which 

• is student-centered and participatory; 

• uses appropriate formal and/or informal assessment meth- 
ods; 

• is used to record gains in literacy development as well as in 
related areas such as independence and self-esteem; 

• provides information that can be used by both tutors and stu- 
dents to review progress and make further lesson plans; 

• is done in such a way that results can be aggregated across 
students and used in program evaluation. 
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Good practice statement 


Standard 


d)The student receives 
individualized instruc- 
tion that builds on his or 
her existing skills. The 
program provides 
options for learning that 
respond to the student’s 
learning styles and 
emerging needs and 
interests. 


i. The program uses the results of the initial interview and assess- 
ment to match the student with a tutor. 

ii. The tutor and student meet at least once per week. 

iii. The tutor uses instructional strategies and content that are appro- 
priate for the student's goals, skills, and interests. 

iv. The tutor uses instructional methods that enable the student to 
apply the skills being learned. 

v. A student leaving the program is encouraged to return if or when 
his or her future goals include further literacy development. 


e)Program staff offer regu- 
lar and timely support 
and guidance to the stu- 
dent. Programs also pro- 
vide appropriate 
referrals for students 
needing to address 
issues that may impose 
barriers to learning. 


i. Staff members provide follow-up during the first month of tutor- 
ing and continue to be available when the student has questions or 
concerns. 

ii. Staff contact each student at least once every three months. 

iii. The program helps students find needed support services, both 
within and outside the program. 

iv. Current information on other education and training programs is 
available and students are assisted in using it. 


f) The literacy program cel- 
ebrates the student’s 
efforts and achieve- 
ments. 


i . Both staff and tutor give the student frequent praise and positive 
reinforcement. 

ii. The program honours each student’s efforts and achievements, in 
an appropriate manner, at least once a year. 
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4. QUALITY TUTORING 



Good practice statement 


Standard 


a) The program recruits 
tutors who are willing to 
help students achieve 
their literacy-related 
goals. 


i. In recruiting tutors, the program clearly outlines the qualifications 
required and screens potential tutors to ensure they meet those 
qualifications. 

ii. The program has clearly written and realistic job descriptions for 
its tutors. 

iii. An interview is used primarily to discuss the tutor’s expectations 
and to explain the program’s requirements. 


b)The program offers 
tutors relevant initial and 
on-going training. 


i. The program provides initial tutor training in at least the following 
areas: 

• introduction to literacy 

• learner-centred approaches, methods and strategies in 
reading, writing, and/or math 

• available resources and support 

• tutor roles and responsibilities. 

ii. The program assesses the on-going training needs of its tutors and 
provides training to meet those needs. 

iii. The program assesses the effectiveness of its tutor training. 


c)The program supports 
and guides tutors as they 
carry out their responsi- 
bilities. 


i. Staff maintain contact with tutors who are not yet matched with a 
student. 

ii. Staff members provide follow-up during the first month of tutor- 
ing and continue to be available when the tutor has questions or 
concerns. 

iii . Staff contact each tutor at least once every 3 months. 

iv. Staff help tutors with resource materials as needed. 

v. Staff help tutors evaluate the effectiveness of their tutoring and 
support them in improving their techniques as necessary. 

vi. The program provides opportunities for tutors to network. 


d)The program recognizes 
the contributions of its 
tutors. 


i. Staff informally acknowledge the efforts of tutors on a regular 
basis by showing them that they are valued and appreciated. 

ii. At least once a year, the program explicitly recognizes the contri- 
butions of its tutors. 
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5. SOUND PROGRAM ADMINISTRATION 



Good practice statement 


Standard 


a)Ultimate authority and 
responsibility for the 
program are clearly 
defined. 


i. Where a program has a hosting authority, the relationship between 
it and the literacy program is clearly defined and documented. 

ii. Lines of authority within the program are clearly defined and doc- 
umented. 

iii .Responsibility for all aspects of the program is clearly defined and 
documented. 


b)The literacy program is 
fully accountable for all 
the funding it receives. 


i. The program follows the funding guidelines provided by its 
funder(s). 

ii. The program's fiscal management includes, at minimum, 

• an annual budget that is approved by the board, advisory 
council, or appropriate body, 

• regular reports on the program’s financial status to the 
board or advisory council, and to the coordinator if he/ 
she does not manage the budget, and 

• an annual review or audit. 


c)The program hires staff 
who have appropriate 
education, experience, 
and personal skills to 
carry out their duties in 
the context of their com- 
munity. 


i. The broad qualifications looked for in paid staff, depending on the 
position they are being hired for, include the following: 

• adult education or training 

• teaching reading and writing 

• instructional planning 

• program planning and administration 

• volunteer management 

ii. The program clearly outlines the specific qualifications required 
by its paid staff. 

iii. The program uses comprehensive, clearly written, and realistic job 
descriptions for all paid staff. 


d)The program recruits 
volunteers who will 
respect its philosophy 
and who will work to 
achieve the program’s 
goals. 


i. The program’s recruitment strategy includes using clearly outlined 
qualifications and personal interviews to screen potential volun- 
teer staff. 

ii. The program uses current, clearly written, and realistic job 
descriptions for volunteer staff positions. 
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Good practice statement 


Standard 


e)The program provides 
the time and fiscal 
resources necessary for 
all paid and volunteer 
staff to receive appropri- 
ate training. 


i. The program provides all new staff with an orientation to the pro- 
gram's policies, operations, and context. 

ii. The program identifies the on-going professional development 
requirements of staff. 

iii. The program ensures that paid staff have access to relevant initial 
and on-going training. Depending on their roles in the program, 
staff training includes the following areas: 

• teaching reading, writing, and math 

• volunteer management 

• assessment techniques and tools 

• program and office management 

• building community partnerships 

• current literacy trends and research 

iv. The program ensures that volunteer staff have access to initial and 
on-going training in areas relevant to their duties. 

v. The program monitors the effectiveness of training for paid and 
volunteer staff. 

vi .Through its links with other literacy organizations at regional and 
provincial levels, the program provides networking opportunities 
for its paid staff at least twice a year. 


f)The program supports 
and guides volunteers as 
they carry out their 
responsibilities. 


i. The program's expectations of volunteer staff are clearly defined, 
documented, and communicated. 

ii. The program provides volunteer staff with the resources they need 
to complete their tasks. 

iii. The program has a means of keeping all volunteers informed. 


g)The program has a strict 
policy of confidentiality 
regarding students and 
volunteers. 


i. The program has practices in place to protect the confidential 
nature of participant records. 

ii. The program clearly communicates to volunteers, students, and 
paid staff the program’s expectations regarding confidentiality for 
all program participants. 


h) An appropriate amount 
of time and resources are 
given to each area of 
program operation. 


i. The program’s plans for recruiting students and volunteers take 
into consideration the space and resources available to the pro- 
gram. 

ii. The coordinator organizes staff time so that both the participants' 
needs and the program's administrative demands are met within 
the time available. 
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Good practice statement 


Standard 


i) The program recognizes 
the contributions of staff 
and volunteers. 


i. The program uses a sufficient amount of its funding to ensure fair 
and equitable staff wages. 

ii. Staff informally acknowledge the efforts of volunteers on a regular 
basis by showing them that they are valued and appreciated. 

iii. At least once a year, the program explicitly recognizes the contri- 
butions of its volunteers. 


j)The literacy program 
maintains accurate, up- 
to-date records and pro- 
gram-related documents. 


i. The program has policies and procedures on file that apply to paid 
staff, volunteers, and students, and reviews them annually. 

ii. The program keeps up-to-date, relevant participant records. 

in.' Tutors submit records of the tutor-student pairs' activities as 
required. 

iv. Assessment results are recorded in a way that demonstrates the 
student’s progress toward his or her goals and the learning out- 
comes achieved. 

v. The program keeps accurate, current financial records on file. 

vi .Program staff prepare reports as needed for the purposes of being 
accountable and of informing the community. 

vii.The program has a comprehensive, annually reviewed records 
management plan that outlines 

• which records to keep and for how long, 

• who has access to which records, and 

• proper disposal of records. 


k)The literacy program has 
suitable office and tutor- 
ing space. Learning 
resources, office equip- 
ment and supplies meet 
the program’s needs. 


i. The program provides enough resource materials for every tutor- 
student pair, and makes the materials available to all participants. 

ii. Resource materials are relevant to the experience, background, 
skill levels, and interests of students and tutors. 

iii. The program reviews its resource materials annually to ensure that 
they meet the participants' needs. 

iv. The program has access to facilities that are safe and inviting for 
all participants, and that are adequate both for administrative tasks 
and for working with participants. 

v. The program ensures that staff have access to the office and 
instructional equipment necessary to carry out their responsibili- 
ties. 
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6. CONTINUOUS PROGRAM DEVELOPMENT 



Good practice statement 


Standard 


a)The literacy program has 
a regular planning and 
evaluation cycle. 


i. The program has documented plans for the following program 
areas, and reviews those plans over a three-year cycle: 

■ assessment of community needs 

■ community partnerships 

■ program promotion and public relations 

■ initial, on-going, and exit assessment of students 

■ student goal setting 

■ student support 

■ tutor/volunteer recruitment, training, and support 

■ staff training and support 

■ program evaluation 

ii. The program develops a long-range plan for future growth and 
direction and reviews that plan at least every three years. 

iii. The program is informed of and incorporates relevant develop- 
ments in literacy practice into its program planning. 

iv. The program uses the results of program evaluation to set goals for 
the next year. 

v. Program evaluation tools are compatible with the program's phi- 
losophy, goals, and delivery methods. 

vi. The program uses as its key indicator of effectiveness the progress 
that students make toward their goals. 

vii. The program uses a variety of measures, both quantitative and 
qualitative, throughout the program cycle, to document its success. 

viii. Program evaluation includes annual performance reviews and 
self-evaluation of paid staff. 


b)The program gives all 
participants the opportu- 
nity to be involved in 
planning and evaluation. 


i. Program planning includes giving participants the opportunity to 
take part in setting program goals and objectives. 

ii. The program’s evaluation strategies include giving participants the 
opportunity to take part in evaluating the program’s effectiveness. 

iii. The program encourages participants to give feedback on the pro- 
gram at any time. 



Standards for Volunteer Literacy Programs in Alberta 



Appendix A 

Project 

Background 



There were four general factors that led to the Alberta Literacy Program 
Standards Project and to the development of this document: the growing 
maturity of literacy programs in this province, the need for a more sys- 
tematic way to demonstrate the value of our programs, the results of a 
pilot project in program evaluation, and the need for formal accountabil- 
ity. 

Since the early 1980’s, literacy programs in Alberta have grown both in 
number and sophistication. While all coordinators are faced with the day- 
to-day challenge of recruiting tutors, matching students, and finding sup- 
port for their programs, many practitioners are looking beyond the imme- 
diate activities of their programs to the larger picture of how programs 
can deliver even better services through professional and organizational 
development. This desire to move literacy practice in Alberta forward is 
one aspect of the project background. 

ALPS also came about as a result of coordinators wanting to demonstrate 
in a more systematic way the value that their programs add to their com- 
munities. Through the use of program standards, practitioners can dem- 
onstrate the value of their programs to participants, to their communities, 
and to current and potential partners. 

An express need for ALPS was made evident as a result of a program 
evaluation project undertaken by Prospects Literacy Association in Edm- 
onton. In the Pilot Project in Program Evaluation (1994-95), four com- 
munity literacy programs piloted two evaluation tools to determine their 
feasibility for use in Alberta. While both tools proved to be useful to 
some degree, the project results strongly identified a need to develop 
evaluation tools specific to volunteer literacy programs in this province. 
In order to develop effective Alberta- specific evaluation tools, the literacy 
community first had to know what it was that needed to be evaluated. In 
other words, it needed to develop good practice statements and then stan- 
dards for programs. 

The fourth factor leading to the ALPS Project is the increasing expecta- 
tions for accountability and evaluation on the part of funders. As part of 
its planning activities, the Alberta Department of Advanced Education 
and Career Development (AE&CD) has included assessment and 
increased accountability as part of its broad goals for adult learning. The 
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Department will require providers to measure and report on performance 
through an accountability framework. This will be used to advise Alber- 
tans of results achieved in publicly-funded programs, and to ensure that 
providers have met appropriate standards of quality to protect the learner. 

All programs funded through Community Programs of AE&CD, includ- 
ing literacy programs, are also impacted by this focus on accountability 
and performance measurement. The Department’s work in the area 
includes developing an accountability framework, program performance 
indicators, and reporting and feedback structures for programs under its 
mandate. Volunteer literacy programs are in something of a unique posi- 
tion in that the Association of Literacy Coordinators of Alberta has 
received “external” funding, through the National Literacy Secretariat, to 
develop its own accountability measures and fit these into the larger 
framework of the provincial government’s initiative on performance mea- 
surement and accountability. 

As described elsewhere in this document, the primary resources for 
developing these standards were the literacy coordinators of Alberta. In 
order to ensure that nothing essential had been missed, other models and 
initiatives in good practice and standards were reviewed. This included 
work in literacy, English as a Second Language, workplace literacy, and 
education, both within Alberta and across Canada and the United States. 
(See the bibliography in Appendix C for a list of resource materials.) 
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Appendix B 

Consultation 

Process 



ALPS is based on the belief that, in order to be useful and relevant, stan- 
dards must be developed from the program level up; that is, that they 
must accurately reflect what literacy coordinators in Alberta see as the 
essential elements of good practice. The primary source of information 
for this work, therefore, has been the membership of the Literacy Coordi- 
nators of Alberta. ALPS also consulted with its Advisory Committee, 
which includes literacy coordinators and representatives from related 
fields who have expertise in evaluation and standards, with representa- 
tives of Advanced Education and Career Development, and with other 
standards initiatives. 



i Consult with 
Community Programs 
to ensure ALPS will 
mesh effectively with 
the government’s 
initiative on 
accountability. 




Develop a draft of 
standards based on input 
from literacy coordinators 
during the good practice 
consultations, and on other 
standards initiatives . 



1 



Using the LCA regional 
network, hold consultations 
with literacy coordinators to 
obtain feedback on draft 
standards . 



1 



Using a mail-out process, 
obtain feedback from 
literacy coordinators on the 
revised set of standards. 



i 



Ask LCA standard 
membership to approve the 
standards as benchmarks for 
use in developing and 
evaluating volunteer literacy 
programs. 



Consult with the 
ALPS Advisory 
Committee on 
overall consultation 
process and the 
various drafts of 
standards 
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Appendix C 

Bibliography 




(We have omitted the bibliography for the “Standards for Volunteer Lit- 
eracy Programs in Alberta” document as all of the items are included in 
the bibliography of the main document.) 




Standards for Volunteer Literacy Programs in Alberta 




176 



^ Appendix F 


Bibliography and recommended reading 


w 

# 


Anderson, Stephen E. (1987). Evaluation Manual for Community-Based 
Training Programs. Toronto: George Brown Community College. 

Best Practice Guidelines for Adult ESL/LINC Programming and Instruc- 
tion in Alberta. (1995). Edmonton: Alberta Teachers of English as a 
Second Language. 

Bhola, H.S. (1990). Evaluating “Literacy for Development” Projects, 
Programs, and Campaigns. Hamburg: UNESCO Institute for Educa- 
tion and the German Foundation for International Development. 

Bogdan, Robert C. and Sari K. Biklen. (1992). Qualitative Research for 
Education. Second Edition. Boston: Allyn and Bacon. 

Canadian Language Benchmarks: English as a second language for 
adults, English as a second language for literacy learners. (Working 
document). (1996). Ottawa: Citizenship and Immigration Canada. 

The Council for Literacy Education, Assessment and Recognition 
(CLEAR). (1991). A proposal by the Practitioner Training Special Inter- 
est Group of the Ontario Literacy Coalition. 

“Creating Practical Knowledge Through Action Research: Posing Prob- 
lems, Solving Problems, and Improving Daily Practice.” Quigley, B. 
Allan and Gary W. Kuhne, Eds. New Directions for Adult and Continu- 
ing Education. #73, Spring 1997. 

Deming, W. Edward. (1986). Out of the Crisis. Cambridge MA: Massa- 
chusetts Institute of Technology Centre for Advanced Engineering 
Study. 

Dickson, Graham and Susan Doyle, Ray Latta. (n.d.). Developing Indica- 
tors and Standards: Choosing Appropriate Measures and Targets of 
Education Performance. Vancouver: B.C. School Superintendents' 
Association. 

Doing It Right! A Needs Assessment Workbook. (1988). Edmonton: Edm- 
onton Social Planning Council. 

Exemplary Practice in Manitoba: Models of Quality in Literacy Pro- 
gramming. (1995). Winnipeg: Literacy and Continuing Education 
Branch, Manitoba Education and Training. 

Fowler, Floyd J., Jr. (1984). Survey Research Methods. Applied Social 
Research Methods Series, Volume 1. Beverly Hills CA: Sage Publica- 
tions. 


# 




Setting the Compass 


page 174 


0 

ERJC 


177 



177 




Framework and Quality Standards for Adult Literacy Education in 
Ontario. (1994). Toronto: Ontario Training and Adjustment Board. 

“Getting the Standards” in ALBSU Basic Skills, Winter 1995. 

Gilbert, Sid. (1994). “The Search for Education Indicators” in Education 
Quarterly Review 1(4): 44-52. 

Guidelines for Program Development, (n.d.). Winnipeg: Literacy and 
Continuing Education Branch, Manitoba Education and Training. 

Holt, Daniel D., Ed. (1994). Assessing Success in Family Literacy 
Projects: Alternative Approaches to Assessment and Evaluation. Wash- 
ington DC: Center for Applied Linguistics and Delta Systems Co., Ltd. 
ED 375 688. 

How to Evaluate Program Impacts in Not-For-Profit Organizations. 
(Course materials). (1998). Banff: The Banff Centre for Management. 
(1-800-590-9799) 

“In Search of Quality Standards” in ALBSU Basic Skills, Spring 1994. 

Joint Committee on Standards for Educational Evaluation with James R. 
Sanders, Chair. (1994). The Program Evaluation Standards: How to 
Assess Evaluations of Educational Programs. Thousand Oaks CA: 
Sage Publications. 

Jurmo, Paul and Sue Folinsbee. (1994). Collaborative Evaluation: A 
Handbook for Workplace Development Planners. Don Mills: ABC 
Canada Literacy Foundation. 

Kazemek, Francis E. (1988). “Necessary Changes: Professional Involve- 
ment in Adult Literacy Programs.” Harvard Educational Review. Vol- 
ume 58, Number 4. 

LCA Skill Profile: Self Assessment for Adult Literacy Coordinators . 
(1995). Calgary: Association of Literacy Coordinators of Alberta. 

Lindman, Murray. (1995). Measure for Measure: Accountability and 
Performance Measurement of Adult Development Programs in Alberta 
(Draft). Edmonton: Adult Development Branch, Alberta Advanced 
Education and Career Development. 

MacLeod, Carol. (1995). Principles of Good Practice in Workplace/ 
Workforce Education: A Report on the Think Tank. Don Mills ON: 
ABC Canada. 

Manitoba Adult Literacy Programming Good Practice Guide. (1995- 
1996). Winnipeg: Literacy and Continuing Education Branch, Mani- 
toba Education and Training. 



Setting the Compass 




page 175 



178 





Measuring Program Outcomes: A Practical Approach. (1996). Alexan- 
dria VA: United Way of America. 

Merriam, Sharan B. (1988). Case Study Research in Education: A quali- 
tative approach. San Francisco CA: Jossey-Bass Inc. 

Merriam, Sharan B. and Edwin L. Simpson. (1989). A Guide to Research 
for Educators and Trainers of Adults. Malabar FL: Robert E. Kneger 
Publishing Company. 

Model Indicators of Program Quality for Adult Education Programs. 
(1992). Office of Vocational and Adult Education, U.S. Department of 
Education. 

National Quality Standards for Volunteer Literacy Programs. (1996). 
Syracuse NY: Laubach Literacy Action. 

Organizing Adult Literacy and Basic Education in Canada: A Policy and 
Practice Discussion Document, (n.d.). Ottawa: Movement for Cana- 
dian Literacy. 

Padak, Nancy and Gary M. Padak. (1991). “What Works: Adult Literacy 
Program Instruction” in Journal of Reading 34 (5): 374-379. 

Patton, Michael Q. (1990). Qualitative Evaluation and Research Meth- 
ods. Second Edition. Newbury Park CA: Sage Publications. 

Perrin, Burt. Outcome Measures: The Greatest Thing Since Sliced 
Bread? or An Impossible Task? Workshop notes, Edmonton, October 7, 
1997. 

Porteous, Nancy L., Barbara J. Sheldrick, and Paula J. Stewart. (1997). 
Program Evaluation Tool Kit: A blueprint for public health manage- 
ment. Ottawa: Ottawa - Carlton Health Department. 

Quality Standards for Adult Literacy : Discussion Guide . ( 1 993) . Toronto: 
Literacy Branch, Ministry of Education and Training. 

Quality Standards for Basic Skills. (1992). London, England: Adult Liter- 
acy and Basic Skills Unit. 

Salant, Priscilla and Don A. Dillman. (1994). How to Conduct Your Own 
Survey. Toronto: John Wiley and Sons, Inc. 

Sanders, Maureen and Mamie Schaetti, Candice Jackson, Karen Man- 
weiler, Pat Campbell. (1995). A Pilot Project in Program Evaluation. 
Edmonton: Prospects Literacy Association. 

Thomas, Audrey. (1989). Adult Literacy Volunteer Tutor Program Evalu- 
ation Kit. Victoria B.C.: Ministry of Advanced Education and Job 
Training and Ministry Responsible for Science and Technology. 


Setting the Compass 


page 176 



o 



179 



Thompson, Barbara. (1995). A Review of the Good Practice Guide with 
Recommendations. Winnipeg: Literacy and Continuing Education 
Branch, Manitoba Education and Training. 

Ysseldyke, James E. and Martha L. Thurlow. (1993). Self-Study Guide to 
the Development of Educational Outcomes and Indicators. Minneapo- 
lis: National Center on Educational Outcomes. 

Ysseldyke, James E., Martha L. Thurlow, Cheri J. Gilman. (1993). Edu- 
cational Outcomes and Indicators for Students Completing School. 
Minneapolis: National Center on Educational Outcomes. 

Ziegler, Suzanne. (1996). The Effectiveness of Adult Literacy Education: 
A Review of Issues and Literature Related to Outcome-Based Evaluation 
of Literacy Programs. Toronto: Ontario Literacy Coalition. 



0 



Setting the Compass 



180 



page 177 




Notes 



Instructions: Use this page to keep track of the evaluation, in terms of 
what worked well, and what could be changed or improved during the 
next evaluation. 



( Photocopy and save original) 
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